FCAnet: A novel feature fusion approach to EEG emotion recognition based on cross-attention networks

Abstract

Recognition of human emotions from complex, diverse, and variable electroencephalogram (EEG) signals based on feature fusion strategies is gaining significant attention in affective computing. However, existing strategies employ a unified learning paradigm to integrate multiple features, and they lead to the neglect of critical local patterns and intricate relationships among features. Furthermore, most of models are generally designed to be complex and difficult to leverage and interpret in practical applications. To address these issues, FCAnet, a novel generic cross-attention-based feature fusion approach is prpopsed. FCAnet views the embedding space of multiple features as interactive patterns and utilizes cross-attention mechanisms to analyze both spatial correlations and discrepancy information simultaneously. Specifically, a dual-branch feature extraction module (DBFE) is first designed to effectively capture the 3D differential entropy (DE) and 3D power spectral density (PSD) feature maps of EEG. Secondly, a cross-attention feature fusion network (CAFFN) integrates a designed discrepancy information injection block (DIIB) with a common information injection block (CIIB) unit, facilitating significant interaction between different features. Finally, a time augmentation block (TAB) is employed to recover information lost from high-level representations, reusing discriminative temporal feature maps. Experimental results on the datasets DEAP, SEED, SEED-IV, and MPED demonstrate that the proposed FCANet outperforms state-of-the-art methods.