Detection of error in static and dynamic visual stimulation via electroencephalogram and eye-tracking systems

Human responses such as electroencephalogram (EEG), eye-tracking, and heart rate have been studied for error detection during visual stimulation, often in controlled settings with single-target fixation. This study delves into constructing machine learning (ML) models for binary error classification across diverse visual stimulation conditions, including static, dynamic, and single or multiple targets, using EEG and eye-tracking data. When constructing these models, using gaze fixation data for epoch extraction can enhance the ability to extract salient, stimulus-induced responses from EEG and eye-tracking data. These features can be strongly associated with changes in visual stimulation. Among 30 ML models tested, the best-performing ML models built on a personalized approach consistently achieved over 90 % accuracy across conditions. For feature importance, we integrate a repetition approach with the Boruta SHapley Additive exPlanations (BorutaSHAP) algorithm to enhance the legitimacy of key feature selection. Feature analysis revealed distinct patterns, e.g., eye-tracking features like log energy entropy being particularly prominent under dynamic conditions, EEG features from the delta, and theta bands being significant across all conditions. Interestingly, an increase in the number of visual targets led to a reduction in the importance of EEG features, especially during dynamic stimulations. These insights have the potential to enhance the ML models through tailored feature selection. While this study acknowledges certain limitations concerning real-time applicability, generalizability, etc, the novelty of our models p[presents opportunities for various applications in human-computer/robot interaction (HCI/HRI), monitoring systems, rehabilitation systems, assistive technologies for individuals with limited mobility and driving, etc.