A stochastic gradient tracking algorithm with adaptive momentum for distributed optimization

Abstract

In this paper, we study distributed optimization problems where each node owns a local convex cost function calculated as the average of multiple constituent functions, and multiple nodes collaborate to minimize the finite sum of these local functions. Reviewing existing work, distributed optimization methods with adaptive momentum that consider reducing computation costs have not yet been explored. To this aim, we propose a gradient tracking stochastic distributed optimization algorithm with adaptive momentum, called GTSADAM. GTSADAM combines the distributed adaptive momentum method for faster convergence with the variance reduction mechanism to reduce computation costs. We provide a convergence analysis indicating that, under certain step size conditions, GTSADAM achieves linear convergence in the mean to the exact optimal solution when each constituent function is strongly convex and smooth. Moreover, GTSADAM maintains the acceleration efficiency of adaptive momentum while minimizing computation costs, which is confirmed by numerical simulations, and its performance is better than that of existing methods.