How to Trap a Gradient Flow

Abstract

SIAM Journal on Computing, Volume 53, Issue 4, Page 803-824, August 2024.
Abstract. We consider the problem of finding an [math]-approximate stationary point of a smooth function on a compact domain of [math]. In contrast with dimension-free approaches such as gradient descent, we focus here on the case where [math] is finite, and potentially small. This viewpoint was explored in 1993 by Vavasis, who proposed an algorithm which, for any fixed finite dimension [math], improves upon the [math] oracle complexity of gradient descent. For example for [math], Vavasis’s approach obtains the complexity [math]. Moreover, for [math] he also proved a lower bound of [math] for deterministic algorithms (we extend this result to randomized algorithms). Our main contribution is an algorithm, which we call gradient flow trapping (GFT), and the analysis of its oracle complexity. In dimension [math], GFT closes the gap with Vavasis’s lower bound (up to a logarithmic factor), as we show that it has complexity [math]. In dimension [math], we show a complexity of [math], improving upon Vavasis’s [math]. In higher dimensions, GFT has the remarkable property of being a logarithmic parallel depth strategy, in stark contrast with the polynomial depth of gradient descent or Vavasis’s algorithm. We augment this result with another algorithm, named cut and flow (CF), which improves upon Vavasis’s algorithm in any fixed dimension.