An Adaptive Gradient Descent Optimization Algorithm Based on Stratified Sampling

Abstract

A necessary part of deep learning is the adjustment of hyperparameters, which is also one of the most expensive parts of deep learning. The current mainstream adaptive learning rate algorithms include AdaGrad, RMSProp, Adam, and AdamW. AdaGrad can adapt to different learning rates for different parameters. However, its adaptive learning rate is monotonically reduced, which will lead to a weak ability to update parameters in the later stage. Although RMSProp, Adam, and AdamW solved the problem of gradually decreasing the adaptive learning rate of AdaGrad, they all introduced a hyperparameter—the momentum coefficient. In this paper, a new optimization algorithm SAdam is proposed. SAdam uses the stratified sampling technique to combine the windows with fixed first-order and second-order gradient information, which not only solves the problem that the adaptive learning rate of AdaGrad is constantly decreasing but also does not introduce additional hyperparameters. Moreover, experiments show that the test accuracy of SAdam is no less than Adam.