FER-VMamba: A robust facial expression recognition framework with global compact attention and hierarchical feature interaction

Abstract

Facial Expression Recognition (FER) has broad applications in driver safety, human–computer interaction, and cognitive psychology research, where it helps analyze emotional states and enhance social interactions. However, FER in static images faces challenges due to occlusions and pose variations, which hinder the model’s effectiveness in real-world scenarios. To address these issues, we propose FER-VMamba, a robust and efficient architecture designed to improve FER performance in complex scenarios. FER-VMamba comprises two core modules: the Global Compact Attention Module (GCAM) and the Hierarchical Feature Interaction Module (HFIM). GCAM extracts compact global semantic features through Multi-Scale Hybrid Convolutions (MixConv), refining them with a Spatial Channel Attention Mechanism (SCAM) to improve robustness against occlusions and pose variations. HFIM captures local and global dependencies by segmenting feature maps into non-overlapping partitions, which the FER-VSS module processes with Conv-SCAM-Conv for local features and Visual State-Space (VSS) for global dependencies. Additionally, self-attention and relation-attention mechanisms in HFIM refine features by modeling inter-partition relationships, further improving the accuracy of expression recognition. Extensive experiments on the RAF and AffectNet datasets demonstrate that FER-VMamba achieves state-of-the-art performance. Furthermore, we introduce FSL-FER-VMamba, an extension of FER-VSS optimized for cross-domain few-shot FER, providing strong adaptability to domain shifts. https://github.com/SwjtuMa/FER-VMamba.git.