A New Approach to Generation and Analysis of Gradient Methods Based on Relaxation Function

Abstract

The goal of this article is to inform English-speaking computer mathematicians about some optimization technique results obtained in publications [1-3]. For the class of matrix gradient methods a new concept of relaxation function is suggested. This concept allows to evaluate an effectiveness of each gradient optimization procedure, and to synthesize new methods for special classes of non-convex optimization problems. According to suggested approach, it is possible to build relevant gradient method for any given relaxation function. The theorem of relaxation conditions for each matrix gradient method is proven. Based on the concept of relaxation functions it is given geometric interpretation of relaxation properties of gradient methods. According to this interpretation it is possible to build a relaxation area, and to evaluate the speed of objective function values decreasing. The analysis of classical matrix gradient schemes such as simple gradient method, Newton's methods, Levenberg-Marquardt method is given. It is shown that relaxation function and its geometric interpretation give almost full information about properties and capabilities of relevant gradient optimization methods.