Activation of connections to accelerate the learning in recurrent back-propagation

Abstract

A method of accelerating learning in recurrent neural networks is described. To activate and use connections, a complexity term defined by an equation was added to a standard quadratic error function. In experiments, a method was used in which a derivative of the complexity term was normally effective for positive connections, while negative connections were pushed toward smaller values. Thus, some connections were expected to be activated and were large enough to speed up learning. It was confirmed that the complexity term was effective in increasing the variance of the connections, especially the hidden connections. It was also confirmed that eventually some connections, especially some hidden connections, were activated and were large enough to be used in speeding up learning.<>