Eigenvalue Analysis on Singularity in RBF networks

Abstract

It has long been observed that strange behaviors happen in the gradient learning process of neural networks including multilayer perceptrons (MLPs) and RBF networks because of the singularities arisen from the symmetric structure in these models. The learning behaviors nearby are crucially dependant on the stability of the singularity. For RBF networks, this paper analyzes the stability by investigating the eigenvalues of the Hessian matrix on the overlap singularities. We show that the overlap singularity is a partially stable critical line, and there is only one nonzero eigenvalue on the singularity. The influence of the teacher parameters and initial conditions on eigenvalues is also discussed.