An explainable eye-tracking-based framework for enhanced level-specific situational awareness recognition in air traffic control

Abstract

Situational awareness (SA) recognition is essential for air traffic controllers (ATCOs) to ensure operational safety in human-AI collaborative environments. The existing studies have primarily focused on overall SA assessment, neglecting its three distinct levels: perception (SA1), comprehension (SA2), and projection (SA3). This study presents an explainable eye-tracking-based three-phase framework for SA recognition. In Phase 1, an unsupervised learning approach was employed to annotate SA levels from behavioral data. Phase 2 involved statistical analysis to extract salient eye-tracking features associated with each SA level. In Phase 3, an ensemble model was developed by integrating the most effective classical algorithms to perform level-specific SA recognition with enhanced robustness and accuracy; SHAP (SHapley Additive exPlanations) values were further employed to interpret feature contributions for the best-performing model at each SA level. To validate the proposed framework, a simulated air traffic control (ATC) radar monitoring experiment incorporating three-level SA-probe tests was conducted with 18 participants. Five-fold cross-validation assessed overall model performance, while Leave-One-Subject-Out (LOSO) evaluated its generalizability across individuals. The ensemble model achieved consistently high accuracy across all SA levels under both evaluation strategies. SHAP analysis highlighted fixation duration, fixation count, and saccade count as key features, with their contributions varying by SA level. These findings demonstrate the need for level-specific SA recognition and lay the foundation for accurate SA monitoring in ATC and other high-risk domains, improving model transparency and interpretability for enhanced operational safety.