Visually-Inspired Multimodal Iterative Attentional Network for High-Precision EEG-Eye-Movement Emotion Recognition.

Abstract

Advancements in artificial intelligence have propelled affective computing toward unprecedented accuracy and real-world impact. By leveraging the unique strengths of brain signals and ocular dynamics, we introduce a novel multimodal framework that integrates EEG and eye-movement (EM) features synergistically to achieve more reliable emotion recognition. First, our EEG Feature Encoder (EFE) uses a convolutional architecture inspired by the human visual cortex's eccentricity-receptive-field mapping, enabling the extraction of highly discriminative neural patterns. Second, our EM Feature Encoder (EMFE) employs a Kolmogorov-Arnold Network (KAN) to overcome the sparse sampling and dimensional mismatch inherent in EM data; through a tailored multilayer design and interpolation alignment, it generates rich, modality-compatible representations. Finally, the core Multimodal Iterative Attentional Feature Fusion (MIAFF) module unites these streams: alternating global and local attention via a Hierarchical Channel Attention Module (HCAM) to iteratively refine and integrate features. Comprehensive evaluations on SEED (3-class) and SEED-IV (4-class) benchmarks show that our method reaches leading-edge accuracy. However, our experiments are limited by small homogeneous datasets, untested cross-cultural robustness, and potential degradation in noisy or edge-deployment settings. Nevertheless, this work not only underscores the power of biomimetic encoding and iterative attention but also paves the way for next-generation brain-computer interface applications in affective health, adaptive gaming, and beyond.