A State-Space Perspective on the Expedited Gradient Methods: Nadam, RAdam, and Rescaled Gradient Flow

Abstract

Fast gradient-descent algorithms are the default practice in training complex machine learning models. This paper presents the convergence guarantee of two existing adaptive gradient algorithms, Nadam and RAdam, for the first time, and the rescaled gradient flow in solving non-convex optimization. The analyses of all three algorithms are unified by a common underlying proof sketch, relying upon Barbalat's lemma. The utility of another tool from classical control, the transfer function, hitherto used to propose a new variant of the famous Adam optimizer, is extended in this paper for developing an improved variant of the Nadam algorithm. Our experimental results validate the efficiency of this proposed algorithm for solving benchmark machine learning problems in a shorter time and with enhanced accuracy.