Human Inspired Memory Module for Memory Augmented Neural Networks

Abstract

Memory is an essential element in most artificial intelligence systems. Recurrent neural networks and long-short term memories are some examples of deep learning structures that have some memory capabilities. As a new approach for incorporating explicit memory into deep learning systems, memory augmented neural networks have been introduced earlier. The neural Turing machine, as a distinguished and pioneering example, is able to emulate a conventional digital computer but fails in more complex tasks. We propose an external memory module, which is composed of two separate submodules for short- and long-term memories. The long-term memory is structured as a graph, equipped with a read/write mechanism. Being fully differentiable makes this memory system easy to be trained via backpropagation. To show the superiority of the proposed system in complex tasks with longterm dependencies, some experiments are conducted. Our analysis shows that this dual-memory system outperforms the neural Turing machine in terms of convergence speed and loss.