HRDF-MER: Hierarchical feature refinement and cascaded dynamic fusion for multimodal emotion recognition

Abstract

Multimodal Emotion Recognition (MER) is challenged by modality misalignment, shallow temporal cue modeling, and inefficient fusion. This paper proposes HRDF-MER, a framework that integrates hierarchical refinement and cascaded dynamic fusion for more robust emotion recognition. To improve cross-modal alignment and unimodal representation, HRDF-MER introduces a novel Hierarchical Cross-modal Feature Refinement (HCFR) strategy, which integrates Cross-modal Adaptive Alignment (CAA) and Hierarchical Feature Enhancement (HFE). The CAA module employs multi-head cross-attention to construct hierarchical correlation matrices for precise acoustic-text alignment, and the HFE employs a Transformer with cross-modal residual connections to further enhance unimodal representations for robust feature learning. We further propose a Cascaded Multimodal Dynamic Fusion (CMDF) strategy, where a cross-attention encoder captures fine-grained inter-modal dependencies and a gated fusion unit adaptively weights modalities to progressively produce highly discriminative multimodal representations. Moreover, a multi-objective training scheme is proposed to jointly optimize feature alignment and classification by integrating Cross-modal Label Contrastive Loss (CLC Loss) with cross-entropy loss. Extensive experiments on the IEMOCAP and MELD datasets demonstrate that HRDF-MER significantly outperforms state-of-the-art models, while ablation studies further confirm the effectiveness and necessity of each proposed component.