Regularized Nesterov’s accelerated damped BFGS method for stochastic optimization

Abstract

A regularization term is introduced into the approximate Hessian update in the stochastic Broyden–Fletcher–Goldfarb–Shanno (BFGS) method for convex stochastic optimization problems to help avoid near-singularity issues. Additionally, Nesterov acceleration, with a momentum coefficient that dynamically adjusts between a constant value and zero based on the objective function, has been incorporated to enhance convergence speed. However, the inflexibility of the constant momentum coefficient still may lead to overshooting problems, and evaluating objective functions on large datasets is computationally costly. Moreover, this approach presents challenges in solving nonconvex optimization problems. To address these challenges, we propose a regularized stochastic BFGS method that integrates Nesterov acceleration with an adaptive momentum coefficient designed for solving nonconvex stochastic optimization problems. This coefficient adjusts flexibly between a decreasing value and zero based on selected dataset samples, helping to avoid overshooting problems and reduce computational costs. We demonstrated almost sure convergence to stationary points and analyze the complexity. Numerical results on convex and nonconvex classification problems using a support vector machine show that our method outperforms existing approaches.