DiffTST: Diff Transformer for Multivariate Time Series Forecast

Abstract

Deep learning models employing the Transformer architecture have demonstrated exceptional performance in the field of multivariate time series forecasting research. However, these models often incorporate irrelevant or weakly relevant information during the processing of time series, leading to noise. This phenomenon diverts the attention mechanism from crucial features within the time series, thereby impacting the overall forecasting performance. To mitigate this issue, our study introduces DiffTST, which employs a Differential Transformer to enhance the model’s focus on relevant context within the time series, thereby mitigating the influence of noise on forecasting accuracy. The model utilizes independent channels to process time series data, ensuring that each input token contains information from a single channel exclusively. Furthermore, each channel is segmented into multiple patches to facilitate the extraction of local information. Subsequently, the Differential Transformer module is employed to process the sequence features of these patches, alleviating the tendency of Transformer-based models to allocate excessive attention to irrelevant sequence information. Ultimately, the forecast outcomes are derived through a Multi-Layer Perceptron. Our findings indicate that DiffTST achieves higher or comparable long-term forecasting accuracy compared to the current state-of-the-art Transformer-based models. On the main datasets (Weather, Traffic, Electricity), our method reduces MSE by 0.008, 0.087, and 0.023 and MAE by 0.004, 0.069, and 0.025 compared to PatchTST.