Learning Temporal Features With Alternated Similarity and Proximity Attention for Time-Series Prediction

Abstract

Time-series prediction is a fundamental problem in various scientific and engineering domains. Recently, attention-based models have shown great promise in long-term time-series forecasting. However, we prove that vanilla attention is equivalent to a one-step random walk on a bipartite graph between the query and the keys, in which the limited number of walks and simplified graph structure could make it less powerful in capturing complex, high-order featural and temporal dependencies. Inspired by how human brains iteratively reactivate memories through reminding, we propose “Alternated Similarity And Proximity Attention,” or ASAP-attention. ASAP-attention employs a random walk on two concurrent views (graphs) that, respectively, capture the featural similarity and the temporal proximity between time points. In particular, the random walk alternately visits the two graphs, each time remembering the previous probability configuration to build a coherent chain of distributions to retrieve useful historical data. This dynamic interplay between temporal and featural clues enhances the model’s ability to capture implicit and heterogeneous data dependencies without using positional encoding. When incorporating ASAP-attention with encoder-only Transformer architecture, we observed highly promising results against a wide collection of state-of-the-art methods on various benchmark datasets for long time-series forecasts (e.g., weather, electricity, illness, and exchange-rate data). Our source code is available at https://github.com/jychen01/ASAP-attention