A novel Adam approach related to Decoupled Weight Decay (AdamL)

Abstract

The use of optimizers makes it possible to reduce losses during the learning process of a neural network. Currently there are some types of optimizers whose effectiveness has already been proven, an example of this is Adam. Adam is an extension to Stochastic Gradient Decent that makes use of Momentum and Adaptive Learning to converge faster. An interesting alternative to complement the Adam’s work is the addition of weight decay. This is done to decouple the weight decay from the gradient-based update. Some attempts have been developed previously, however its correct operation has not been keenly proven. In this work, a weight decay decoupling alternative is presented and acutely analyzed. The algorithm’s convergence is mathematically verified and its operation too through the use of a Convolutional Encoder-Decoder network and the application of strategies for error reduction. The AdamL operation is verified by the achievement of a proper Temperature Forecast with a percentage error lower than 4.5%. It can be seen too that the forecast error deepens around noon but it does not exceed 1.47°C.