Enhanced inverted transformer: advancing variate token encoding and blending for time series forecasting

Abstract

Recent advancements in channel-dependent Transformer-based forecasters highlight the efficacy of variate tokenization for time series forecasting. Despite this progress, challenges remain in handling complex time series. The vanilla Transformer, while effective in certain scenarios, faces limitations in addressing intricate cross-variate interactions and diverse temporal patterns. This paper presents the Enhanced Inverted Transformer (EiT for short), enhancing standard Transformer blocks for advanced modeling and blending of variate tokens. EiT incorporates three key innovations: First, a hybrid multi-patch attention mechanism that adaptively fuses global and local attention maps, capturing both stable and volatile correlations to mitigate overfitting and enrich inter-channel communication. Second, a multi-head feed-forward network with specialized heads for various temporal patterns, enhancing parameter efficiency and contributing to robust multivariate predictions. Third, paired channel normalization applied to each layer, preserving crucial channel-specific statistics and boosting forecasting performance. By integrating these innovations, EiT effectively overcomes limitations and unlocks the potential of variate tokens for accurate and robust multivariate time series forecasting. Extensive evaluations demonstrate that EiT achieves state-of-the-art (SOTA) performance, surpassing the previous method, the inverted Transformer, by an average of 4.4% in Mean Squared Error (MSE) and 3.4% in Mean Absolute Error (MAE) across five challenging long-term forecasting datasets.