Unsaturated MLP neural networks training algorithm using a piecewise error function and adaptive learning rates

Abstract

Saturation conditions of the hidden layer neurons are a major cause of learning retardation in multilayer perceptrons (MLP). Under such conditions the traditional backpropagation (BP) algorithm is trapped in local minima. To renew the search for a global minimum, we need to detect the traps and an offset scheme to avoid them. We have discovered that the gradient norm drops to a very low value in local minima. Here, adding a modifying term to the standard error function enables the algorithm to escape the local minima. In this paper, we proposed a piecewise error function; i.e. where the gradient norm remained higher than a parameter we used the standard error function, and added a modifying term to the function below this value. To further enhance this algorithm, we used our proposed adaptive learning rate schema. We performed a selection of benchmark problems to asses the efficiency of our proposed algorithm. Compared to previously proposed algorithms, we recorded higher convergence rates, especially in complex problems with complex input-output mapping.