Finite difference and simultaneous perturbation stochastic approximation with fixed step sizes in case of multiplicative noises

Abstract

Simultaneous perturbation stochastic approximation method was shown to be superior over finite difference (Kiefer-Wolfowitz) method in case of unknown but bounded additive measurement noise. This paper is devoted to analysis of the behaviour of these methods in case of multiplicative noise and fixed step sizes. It gives theoretical bounds for the mean squared error and variance after finite number of iiterations for finite difference and simultaneous perturbation methods. The multiplicative noise is present in cost functions in many different fields, and ability to cope with them is a good side of for an optimization method. Fixed step size algorithms are easy to implement and analyze as well as can be used in nonstationary optimization problems. The simulation includes the case when the algorithms' parameters are chosen as theoretically optimal and the case when they are chosen as practically giving the best results after finite number of iterations. Comparative analysis shows similar performance of both methods in terms of mean squared error and slightly better performance of SPSA in terms of variance. Simulation results are provided to illustrate the theoretical contributions.