CLIP-Driver: Multimodal Fusion Method for Driver Emotion Recognition via Event-Image Temporal Alignment

Abstract

Single-eye micro-expression in driving scenarios can reveal the driver’s hidden emotional state, providing crucial evidence for safety monitoring. To address the issues of event–red, green, and blue (RGB) modality gap, insufficient sequence-level dynamic alignment, and redundant-event-frame interference with semantic drift during cross-modal fusion in single-eye-based driver emotion recognition, this article proposes contrastive language-image pretraining (CLIP)-Driver: a multimodal fusion method based on event-image temporal alignment. This method constructs a shared dynamic micro-expression event–image representation space by introducing the CLIP model as a semantic bridge, thereby improving the transferability and semantic coherence of multimodal features. Subsequently, a sequence-level contrastive optimization strategy is employed to capture the dynamic evolution of micro-expressions, reducing the interference of local frame-level alignment noise in temporal representation learning. Finally, a redundancy-aware mechanism combined with consistency optimization adaptively suppresses weakly discriminative event frames and maintains cross-modal semantic consistency during multimodal fusion. Experiments on the publicly available single-eye event-based emotion (SEE) and diverse single-eye event-based emotion dataset (DSEE) datasets demonstrate the effectiveness of CLIP-Driver. Specifically, on the SEE dataset, it achieves a weighted average recall (WAR) of 94.1% and an unweighted average recall (UAR) of 94.6%. On the DSEE dataset, it achieves a WAR of 84.5% and a UAR of 85.4%. These results demonstrate that CLIP-Driver significantly outperforms existing mainstream methods and validate its superior generalization ability and recognition robustness under different lighting conditions.