SANTM: A Sparse Access Neural Turing Machine with local multi-head self-attention for long-term memorization

Abstract

In this paper, we propose a Sparse Access Neural Turing Machine (SANTM) to address long-term memorization challenges in sequence learning. The SANTM integrates a three-level neural controller with external memory: (1) a bottom layer for segmenting inputs into variable-length chunks, (2) a middle layer for short-term memory integration, and (3) a top layer that selectively accesses external memory via a locality-biased multi-head self-attention mechanism based on ChebNet spectral graph convolution. A sparse mask, trained through an ${\ell }_0$-constrained optimization scheme, reduces memory access rates while enabling pre-fetching. Theoretical analysis derives an optimal access rate under idealized conditions. Experiments on sequential image classification (MNIST, CIFAR10), text classification, speaker discrimination, and language modeling (WikiText-103, enwik8) demonstrate SANTM’s superiority over state-of-the-art sequential models. Key results include 95.7% accuracy on permuted MNIST (vs. NTM’s 94.0%), 85.4% on TC-Speech (vs. 79.6% for NTM), and 24.2 perplexity on WikiText-103 (vs. Transformer-XL’s 27.0). The sparse mask reduces FLOPs by 37%–54% compared to traditional NTMs, validating its efficiency.