Deep learning model for patient emotion recognition using EEG-tNIRS data

Abstract

This study presents a novel approach that integrates electroencephalogram (EEG) and functional near-infrared spectroscopy (tNIRS) data to enhance emotion classification accuracy. A Modality-Attentive Multi-Channel Graph Convolution Model (MAMP-GF) is introduced, leveraging GraphSAGE-based representation learning to capture inter-channel relationships. Multi-level feature extraction techniques, including Channel Features (CF), Statistical Features (SF), and Graph Features (GF), are employed to maximize the discriminative power of EEG-tNIRS signals. To enhance modality fusion, we propose and evaluate three fusion strategies: MA-GF, MP-GF, and MA-MP-GF, which integrate graph convolutional networks with a modality attention mechanism. The model is trained and validated using EEG and tNIRS data collected from 30 subjects exposed to emotionally stimulating video clips. Experimental results demonstrate that the proposed MA-MP-GF fusion model achieves 98.77% accuracy in subject-dependent experiments, significantly outperforming traditional single-modal and other multimodal fusion methods. In cross-subject validation, the model attains a 55.53% accuracy, highlighting its robustness despite inter-subject variability. The findings illustrate that the proposed graph convolution fusion approach, combined with modality attention, effectively enhances emotion recognition accuracy and stability. This research underscores the potential of EEG-tNIRS fusion in real-time, non-invasive emotion monitoring, paving the way for advanced applications in personalized healthcare and affective computing.