Combining multi-scale convolutional neural network and Transformers for EEG-Based RSVP detection

Abstract

Rapid serial visual presentation (RSVP) is an effective brain-computer interface (BCI) technique for recognizing target objects. Decoding the subject’s intention from the single-trial electroencephalogram (EEG) signal through a decoding algorithm is the key to RSVP-based BCI. The unavoidable noise and variability between trials in EEG signals lead to low accuracy of EEG-based RSVP detection and low universality of the model. It is necessary to develop an EEG decoding algorithm with robust generalization ability and high recognition accuracy. In this study, we proposed a novel end-to-end model architecture that combines multi-scale spatiotemporal convolutional neural network (CNN) and Transformers. Specifically, the multi-scale CNN is used to capture spatiotemporal features at different scales, while the Transformers are used to extract the most discriminative global information. Experimental results on the RSVP-based benchmark datasets show that the proposed method in this study can achieve higher recognition accuracy compared to the other three advanced methods in both cross-subject and within-subject experiments. The results of fine-tuning experiments using pre-trained models on a new subject show that better results can be obtained in single-subject experiments using only a small amount of data. The experimental results validate the effectiveness of our method and provide a new idea for constructing a feature extraction method with better generalization capability for RSVP-based BCI.