A transformer-based self-supervised pre-training model for time series prediction

Multivariate time series forecasting is ubiquitous in the real world. The performance of prediction model is determined by its representation ability. At present, self-supervised pre-training is the main method to improve the representation ability of prediction models. However, the periodic characteristics of time series are rarely considered in the existing pre-training models. Our experimental study shows that the periodic characteristics of time series have a great impact on the performance of self-supervised pre-training models. To address this issue, we propose a novel self-supervised prediction model, SMformer. SMformer has two distinctive features: (1) A new patch partition Module is innovatively introduced into backbone model transformer using the periodic property of time series. (2) Two pretext tasks, shuffle and mask, are design for the self-supervised pre-training of the model SMformer. We conducted extensive experiments on seven benchmark datasets, and the experimental results demonstrate that SMformer significantly outperforms prior comparison baselines.