CM-FusionNet: A cross-modal fusion fatigue detection method based on electroencephalogram and electrooculogram

Abstract

Mental fatigue detection plays an important role in preventing fatigue-related diseases and reducing traffic accidents caused by mental exhaustion. In this effort, existing studies have presented interesting results by using physiological signals. However, most works focus primarily on single physiological signal like electroencephalography (EEG). To address this gap, we propose an innovative cross-modal fusion method (CM-FusionNet) and conduct a multi-modal study using EEG and electrooculogram (EOG) for mental fatigue detection. Specifically, a variance channel attention (VCA) module is introduced to adaptively learn the optimal weights for each channel. Then, a Transformer fusion module is applied to extract and integrate the global features of EEG and EOG. Finally, we classify mental fatigue using the fused features. With this method, we conduct independent and cross-subject experiments on the public SEED-VIG dataset. The results of multi-modal experiment show an average accuracy of 84.62% and F1-score of 85.25%, an increase by 1.48% in accuracy 2.46% and in F1-score compared to the EOG-only experiment, and increase by 2.88% and 3.92% compared to the EEG-only experiment, respectively. This demonstrates the benefits of incorporating multi-modalities in fatigue detection and highlights the increased accuracy achieved with our CM-FusionNet approach. It also indicates that this method has potential for further exploration in the field of biomedical signal processing.