Self-Supervised Learning-Based Time Series Classification via Hierarchical Sparse Convolutional Masked-Autoencoder

Abstract

In recent years, the use of time series analysis has become widespread, prompting researchers to explore methods to improve classification. Time series self-supervised learning has emerged as a significant area of study, aiming to uncover patterns in unlabeled data for richer information. Contrastive self-supervised learning, particularly, has gained attention for time series classification. However, it introduces inductive bias by generating positive and negative samples. Another approach involves Masked Autoencoders (MAE), which are effective for various data types. However, due to their reliance on the Transformer architecture, they demand significant computational resources during the pre-training phase. Recently, inspired by the remarkable advancements achieved by convolutional networks in the domain of time series forecasting, we aspire to employ convolutional networks utilizing a strategy of mask recovery for pre-training time series models. This study introduces a novel model termed Hierarchical Sparse Convolutional Masked-Autoencoder, “HSC-MAE”, which seamlessly integrates convolutional operations with the MAE architecture to adeptly capture time series features across varying scales. Furthermore, the HSC-MAE model incorporates dedicated decoders that amalgamate global and local information, enhancing its capacity to comprehend intricate temporal patterns. To gauge the effectiveness of the proposed approach, an extensive array of experiments was conducted across nine distinct datasets. The experimental outcomes stand as a testament to the efficacy of HSC-MAE in effectively mitigating the aforementioned challenges.