Micro-expression spotting based on multi-modal hierarchical semantic guided deep fusion and optical flow driven feature integration

Abstract

Micro-expression (ME), as an involuntary and brief facial expression, holds significant potential applications in fields such as political psychology, lie detection, law enforcement, and healthcare. Most existing micro-expression spotting (MES) methods predominantly learn from optical flow features while neglecting the detailed information contained in RGB images. To address this issue, this paper proposes a multi-scale hierarchical semantic-guided end-to-end multimodal fusion framework based on Convolutional Neural Network (CNN)-Transformer for MES, named MESFusion. Specifically, to obtain cross-modal complementary information, this scheme sequentially constructs a Multi-Scale Feature Extraction Module (MFEM) and a Multi-scale hierarchical Semantic-Guided Fusion Module (MSGFM). By introducing an Optical Flow-Driven fusion feature Integration Module (OF-DIM), the correlation of non-scale fusion features is modeled in the channel dimension. Moreover, guided by the optical flow motion information, this approach can adaptively focus on facial motion areas and filter out interference information in cross-modal fusion. Extensive experiments conducted on the CAS(ME)² dataset and the SAMM Long Videos dataset demonstrate that the MESFusion model surpasses competitive baselines and achieves new state-of-the-art results.