Activated hidden connections to accelerate the learning in recurrent neural networks

A method of accelerating the learning in recurrent neural networks is considered. Owing to a possible large number of connections, it has been expected that recurrent neural networks will converge faster. To activate hidden connections and use hidden units efficiently, a complexity term proposed by D.E. Rumelhart was added to the standard quadratic error function. A complexity term method is modified with a parameter to be normally effective for positive values, while negative values are pushed toward values with larger absolute values. Thus, some hidden connections are expected to be large enough to use hidden units and to speed up the learning. From the author's experiments, it was confirmed that the complexity term was effective in increasing the variance of connections, especially hidden connections, and that eventually some hidden connections were activated and large enough for hidden units to be used in speeding up the learning.<>