Exploring EEG and eye movement fusion for multi-class target RSVP-BCI

Abstract

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interfaces (BCIs) enable high-throughput target image detection by identifying event-related potentials (ERPs) in electroencephalography (EEG) signals. Traditional RSVP-BCI systems detect only single-class targets within image streams, limiting their ability to handle more complex tasks requiring multi-class target identification. Multi-class target RSVP-BCI systems are designed to detect multi-class targets in real-world scenarios. However, distinguishing between different target categories remains challenging due to the high similarity across ERPs evoked by different target categories. In this work, we incorporate the eye movement (EM) modality into traditional EEG-based RSVP decoding and develop an open-source multi-modal dataset comprising EM and EEG signals from 43 subjects in three multi-class target RSVP tasks. We further propose the Multi-class Target RSVP EEG and EM fusion Network (MTREE-Net) to enhance multi-class RSVP decoding. Specifically, a dual-complementary module is designed to strengthen the differentiation of uni-modal features across categories. To achieve more effective multi-modal fusion, we adopt a dynamic reweighting fusion strategy guided by theoretically derived modality contribution ratios for optimization. Furthermore, we propose a hierarchical self-distillation module to reduce the misclassification of non-target samples through knowledge transfer between two hierarchical classifiers. Extensive experiments demonstrate that MTREE-Net achieves significant performance improvements, including over 5.4% and 3.32% increases in balanced accuracy compared to existing EEG decoding and EEG-EM fusion methods, respectively. Our research offers a promising framework that can simultaneously detect target existence and identify their specific categories, enabling more robust and efficient applications in scenarios such as multi-class target detection.