MSA-LR: Enhancing multi-scale temporal dynamics in multivariate time series forecasting with low-rank self-attention

Abstract

Accurately forecasting multivariate time series requires effectively capturing intricate temporal dependencies across diverse scales. Existing deep learning models, while promising, often fall short in this regard. Recurrent architectures like LSTMs struggle with long-range dependencies crucial for multi-scale modeling, while standard Transformers, despite employing attention mechanisms, fail to explicitly differentiate the importance of distinct periodicities, treating all time steps within a fixed window with similar relevance. This limitation hinders their ability to leverage the rich hierarchical structure of real-world time series, particularly in long-term forecasting scenarios. This paper introduces MSA-LR (Multi-Scale Self-Attention with Low-Rank Approximation), a novel architecture explicitly designed to capture multi-scale temporal dynamics. MSA-LR leverages a learnable scale weight matrix and low-rank approximations to directly model the influence of different temporal granularities (e.g., hourly, daily, weekly). This approach not only allows for fine-grained control over multi-scale interactions but also significantly reduces computational complexity compared to standard self-attention, enabling efficient processing of long time series. Empirical evaluations on diverse datasets, including electricity load, traffic flow, and air quality, demonstrate that MSA-LR achieves competitive performance compared to state-of-the-art methods, exhibiting notable improvements in long-term forecasting accuracy. Further analysis reveals MSA-LR's ability to discern and leverage periodic patterns at various resolutions, confirming its effectiveness in capturing the rich multi-scale temporal structure of real-world time series data.