L3former: Enhanced multi-scale shared Transformer with Local Linear Layer for long-term series forecasting

Abstract

Long-term time series forecasting is crucial in areas such as energy management and climate modeling. While multi-scale Transformer architectures have demonstrated success in long-term forecasting, they face challenges including high computational complexity and limited effectiveness in multi-scale decomposition and fusion. To address this, we introduce L³former, a Transformer-based multi-scale shared network that integrates Local Linear Layer (L³), Scale-Wise Attention Mechanism (SWAM), and Variable-Wise Feed-Forward Layer (VWFF). L³ is an innovative neural network layer, which independently aggregates temporal information within windows via local linear connections and shares weights across channels, utilizing varying window sizes to construct multi-scale features. SWAM adeptly fuses these multi-scale features by assigning attention weights across different scales. Moreover, all scales share a unified embedding space and backbone network, thereby reducing the complexity of models. Furthermore, VWFF is incorporated into the standard Transformer encoder to mitigate the performance degradation caused by channel independence. On average across nine datasets, L${}^3$former outperforms recent state-of-the-art models, achieving 5.8%–16.7% lower MSE in long-term forecasting tasks.