Perlustration of error surfaces for nonlinear stochastic gradient descent algorithms

Abstract

We attempt to explain in more detail the performance of several novel algorithms for nonlinear neural adaptive filtering. Weight trajectories together with the error surface give a clear understandable representation of the family of least mean square (LMS) based, nonlinear gradient descent (NGD), search-then-converge (STC) learning algorithms and the real-time recurrent learning (RTRL) algorithm. Performance is measured on prediction of coloured and nonlinear input. The results are an alternative qualitative representation of different qualitative performance measures for the analysed algorithms. Error surfaces and the adjacent instantaneous prediction errors support the analysis.