A fast convergence algorithm for BPNN based on optimal control theory based learning rate

Abstract

In this paper, a novel updating law for Backpropagation learning algorithm based on optimal control theory is proposed. The original Backpropagation algorithm composed of learning rate factor (LR). The coefficient in LR is called step size and indicates the rate of algorithm convergence which is selected by trial and error. In original BP the step size is constant. This fixed step size causes important incapabilities such as slow convergence and local minima problem. In Optimal Control Theory Based Learning Rate (OCLR)algorithm the step size is selected adaptively according to optimal control theory that makes Backpropagation learning algorithm convergence much faster than the original BP. To achieve the fastest possible answer, the Backpropagation learning algorithm is modeled as a minimum time control problem and the step size coefficient is considered as input. This consideration results a Bang-Bang control characteristics. The effectiveness of the proposed algorithm is evaluated via two examples. These examples are XOR, 3-bit parity. In all the problems, the proposed algorithm outperforms well in speed and the ability to escape from local minima.