BH-CMA: A cross-modal attention network based on brain-heart collaboration for stress cognitive state decoding

Abstract

Accurate recognition of stress states under different cognitive loads is important for adaptive human computer interaction and physiological stress monitoring. However, unimodal measurements provide incomplete central or autonomic information, while indiscriminate multimodal fusion may obscure the distinct temporal and spatial organization of electroencephalography (EEG) and heart rate variability (HRV). This study proposes BH-CMA, a compact cross-modal attention network based on brain-heart collaboration for stress cognitive state decoding. Modality-specific encoders preserve the frequency, temporal, and scalp spatial organization of EEG and model the temporal dynamics of HRV. EEG representations then guide cross-modal temporal attention to retrieve complementary autonomic information from synchronized HRV. A linear preserving residual classifier further combines stable discriminative information from the original physiological inputs with nonlinear corrections learned by the deep branch. BH-CMA was evaluated using data from 28 participants under four conditions defined by acute stress induction and cognitive load. Leave-one-subject-out (LOSO) evaluation showed that BH-CMA achieved an accuracy of 58.07%, a Macro F1 of 55.54%, and a Macro AUC of 81.74%. Compared with EEG alone, BH-CMA increased mean Macro F1 by 4.15%, with higher Macro F1 in 23 of 28 participants. Masking HRV also reduced Macro F1 in 21 participants, supporting the contribution of HRV information to classification by the multimodal model. BH-CMA achieved significantly higher Macro F1 than its internal linear prediction mode. Gains were larger for cognitive load discrimination, with the largest recall gains in HCS and ASH. BH-CMA provides a compact framework for cross-subject stress cognitive state decoding using coordinated EEG and HRV information.