Fusion Analysis of EEG-fNIRS Multimodal Brain Signals: A Multitask Classification Algorithm Incorporating Spatial-Temporal Convolution and Dual Attention Mechanisms

Abstract

Brain–computer interface (BCI) is an important way of human-computer interaction, with the ability to monitor brain states, and it has become an increasingly significant research direction. Single-modal noninvasive brain signals have limitations, such as low spatial resolution or low temporal resolution, while multimodal brain signal acquisition and processing can overcome these limitations. Electroencephalogram and functional near-infrared spectroscopy (EEG-fNIRS) is a method with advantages in multimodal brain signal processing, but current fusion methods mostly use manual feature extraction or channel selection, which may lead to the loss of important information during the feature extraction or channel selection process in real-time BCI systems. In order to solve this issue, this article proposes an innovative fusion analysis method for EEG-fNIRS multimodal brain signals, using a hybrid algorithm that combines convolutional neural network (CNN) and Attention mechanisms for signal classification. The method first preprocesses the EEG and fNIRS signals separately, then extracts features using spatial-temporal convolutional layers, and finally merges them to obtain the classification results through dual attention calculation. Our method is validated on two publicly available mixed EEG-fNIRS BCI datasets, including three types of experimental tasks that do not involve actual movement: motor imagery (MI), mental arithmetic, and word generation (WG). The accuracy rates for each task reached 92.2% for MI, 98.6% for mental arithmetic, and 95.2% for WG, respectively. These rates have surpassed all the current methods. This indicates that our proposed method achieves better classification performance in non-actual movement classification tasks under the premise of lightweight. The method proposed in this study can be applied to the field of rapid and efficient identification of brain signals.