MSC-transformer-based 3D-attention with knowledge distillation for multi-action classification of separate lower limbs

Abstract

Deep learning has been extensively applied to motor imagery (MI) classification using electroencephalogram (EEG). However, most existing deep learning models do not extract features from EEG using dimension-specific attention mechanisms based on the characteristics of each dimension (e.g., spatial dimension), while effectively integrate local and global features. Furthermore, implicit information generated by the models has been ignored, leading to underutilization of essential information of EEG. Although MI classification has been relatively thoroughly investigated, the exploration of classification including real movement (RM) and motor observation (MO) is very limited, especially for separate lower limbs. To address the above problems and limitations, we proposed a multi-scale separable convolutional Transformer-based filter-spatial-temporal attention model (MSC-T3AM) to classify multiple lower limb actions. In MSC-T3AM, spatial attention, filter and temporal attention modules are embedded to allocate appropriate attention to each dimension. Multi-scale separable convolutions (MSC) are separately applied after the projections of query, key, and value in self-attention module to improve computational efficiency and classification performance. Furthermore, knowledge distillation (KD) was utilized to help model learn suitable probability distribution. The comparison results demonstrated that MSC-T3AM with online KD achieved best performance in classification accuracy, exhibiting an elevation of 2 %-19 % compared to a few counterpart models. The visualization of features extracted by MSC-T3AM with online KD reiterated the superiority of the proposed model. The ablation results showed that filter and temporal attention modules contributed most for performance improvement (improved by 2.8 %), followed by spatial attention module (1.2 %) and MSC module (1 %). Our study also suggested that online KD was better than offline KD and the case without KD. The code of MSC-T3AM is available at: https://github.com/BICN001/MSC-T3AM.