Convolutional channel modulator for transformer and LSTM networks in EEG-based emotion recognition.

Abstract

Electroencephalogram (EEG) signal is receiving much attention from recent studies since it is highly associated with intrinsic emotion. However, EEG signals contain underlying factors of variations across different sessions of the same subject, which make it difficult to learn temporal relationships between successive time steps. To disentangle invariant features, we propose a feature re-weighting mechanism on the extracted EEG features for temporal sequence modeling. Based on this method, our proposed model, called Convolutional Channel Modulator for Transformer and LSTM networks (CCMTL), extracts emotion-related inter-channel correlations using convolution operations and emphasizes important features by generating a channel attention map. This attention map is then used to perform matrix multiplication on the extracted features, which helps the subsequent Transformer to focus on important affective features. Furthermore, the sequential temporal modeling enhances the overall model's capability to understand temporal relationships both in global and sequential contexts. Experimental settings on public EEG emotion datasets demonstrate the superiority of the proposed CCMTL, surpassing six state-of-the-art models. Our code is publicly available at https://github.com/affctivai/CCMTL.