An adaptive stochastic gradient method with variance reduction for smooth optimization problems

Abstract

A Stochastic Gradient Descent(SGD) method has received more attention due to its efficient usage in machine learning. However, this method has some substantial defects, such as the high variance of the stochastic gradients and difficulty in selecting a practically performant learning rate. The former has been mitigated via the introduction of variance-reduced(VR) methods. These approaches reduce the noise due to variance in stochastic gradients; thus improving the convergence of the SGDs. However, these variants with VR property still inherit the latter knotty issue of SGD, where most adopted schemes resort to manual tuning of learning rate in practice. On the otherhand, other improvements with adaptive learning are established under some strong assumptions that may potentially limit their scope of usage. Consequently, this paper introduces a novel adaptive learning rate strategy into a minibatch variant of VR approach named StochaAstic Recursive grAdient algoritHm (SARAH), leading to SARAH-AD. The SARAH-AD algorithm enables an effective, efficient, robust, and automated computation of a suitable learning rate that is independent of the Lipchitz constant of the objective function. Moreover, the theoretical convergence analysis of SARAH-AD is extensively analyzed under a non-strong convexity case satisfying error-bound property. Furthermore, we analyzed the SARAH-AD in a general non-convex setting by requiring only an average L-smoothness, which is weaker than the usual L-smoothness assumption and also without bounded variance assumption. Moreover, we experimentally tested the performance of SARAH-AD on several medium to large-scale real-world datasets. The comparison uses convex and non-convex models with best-tuned algorithms. The proposed algorithm showed consistently better performance than other compared algorithms.