Local linear convergence of stochastic higher-order methods for convex optimization

Abstract

We propose a stochastic higher-order algorithm for solving finite sum convex optimization problems. Our algorithmic framework is based on the notion of stochastic higher-order upper bound approximations of the finite sum objective function. For building such a framework we only require that this bound approximate the objective function up to an error that is p times differentiable and has a Lipschitz continuous p derivative. This leads to a stochastic higher-order majorization-minimization algorithm, which we call SHOM. We show that the algorithm SHOM achieves local linear convergence rate for the function values provided that the finite sum objective function is uniformly convex. Numerical simulations confirm the efficiency of our method.