A new hierarchical temporal memory based on recurrent learning unit

Abstract

ABSTRACT Hierarchical temporal memory is an emerging machine learning technology that aims to model the structural and algorithmic properties of the neocortex. It is particularly suitable for learning and predicting sequential data. However, when dealing with long time series or complex sequences, the accuracy is relatively lower than desired. In this paper, a novel hierarchical temporal memory based on recurrent learning unit is proposed, where a feedback mechanism is involved into the model. The original cell is extended with a recurrent unit to capture long temporal dependencies of synaptic connections between neurons. The temporal pooler algorithm is then modified to adapt to the recurrent learning unit, and the supervised gradient information is combined with the Hebbian synaptogenesis learning rule in speeding up the training. The prototype of the proposed hierarchical temporal memory is implemented and extensive experiments are carried out on two public datasets under various settings. Experimental results show that the proposed model obtains an accuracy increase by up to 32% and a perplexity drop by up to 14% on sequence prediction and text generation tasks, respectively, which indicates the hierarchical temporal memory with recurrent feedback outperforms the original model on sequence learning.