Transforming stroop task cognitive assessments with multimodal inverse reinforcement learning

Abstract

Stroop tasks, recognized for their cognitively demanding nature, hold promise for diagnosing and monitoring neurodegenerative diseases. Understanding how humans allocate attention and resolve interference in the Stroop test remains a challenge; yet addressing this gap could reveal key opportunities for early-stage detection. Traditional approaches overlook the interplay between overt behavior and underlying neural processes, limiting insights into the complex color-word associations at play. To tackle this, we propose a framework that applies Inverse Reinforcement Learning (IRL) to fuse electroencephalography (EEG) signals with eye-tracking data, bridging the gap between neural and behavioral markers of cognition. We designed a Stroop experiment featuring congruent and incongruent conditions to evaluate attention allocation under varying levels of interference. By framing gaze as actions guided by an internally derived reward, IRL uncovers hidden motivations behind scanning patterns, while EEG data — processed with advanced feature extraction — reveals task-specific neural dynamics under high conflict. We validate our approach by measuring Probability Mismatch, Target Fixation Probability-Area Under the Curve, Sequence Score, and MultiMatch metrics. Results show that the IRL-EEG model outperforms an IRL-Image baseline, demonstrating improved alignment with human scanpaths and heightened sensitivity to attentional shifts in incongruent trials. These findings highlight the value of integrating neural data into computational models of cognition and illuminate possibilities for early detection of neurodegenerative disorders, where subclinical deficits may first emerge. Our IRL-based integration of EEG and eye-tracking further supports personalized cognitive assessments and adaptive user interfaces.