Convergence analysis of the heavy-ball random reshuffling method for finite-sum optimization

Abstract

This paper proposes a Heavy-Ball Random Reshuffling (HBRR) method, which extends the standard Random Reshuffling (RR) algorithm by incorporating the classical Heavy-Ball momentum mechanism. We systematically analyze the convergence properties of HBRR under constant step sizes for strongly convex, general convex, and nonconvex finite-sum optimization problems. For strongly convex objectives, we prove linear contraction of the expected squared distance to a step-size-dependent residual neighborhood. For general convex objectives, we derive a sublinear convergence bound for the function-value error of the averaged iterates. For nonconvex objectives, we obtain a complexity bound of the same order as RR. Under the Polyak–Łojasiewicz condition, the expected function-value error contracts linearly to a residual neighborhood and converges exactly when the residual vanishes. The theoretical framework is consistent, as it reduces precisely to the standard RR convergence results when the momentum parameter is zero. Numerical experiments on four benchmark datasets demonstrate that HBRR achieves competitive or superior performance in terms of convergence speed, solution accuracy, and algorithmic stability across all tested scenarios, confirming the effectiveness of incorporating Heavy-Ball momentum into the random reshuffling framework.