AVERFormer: End-to-end audio-visual emotion recognition transformer framework with balanced modal contributions

Abstract

Audiovisual Emotion Recognition (AVER) plays a critical role in various domains, including mental health monitoring, educational interactions, and human-computer interaction. However, existing methods often encounter three main challenges: insufficient feature extraction for each modality, imbalanced modality contributions, and inadequate exploitation of multimodal complementarity. To address these issues, this paper proposes an end-to-end framework called AVERFormer. Specifically, AVERFormer consists of an audio encoder, a visual encoder, and an audiovisual fusion module, with several notable innovations. First, we design a dual-branch audio encoder that extracts multi-frequency emotional features from raw audio waveforms, spectrograms, and Mel-Frequency Cepstral Coefficients (MFCCs). This approach not only captures fine-grained local details but also efficiently processes long-duration audio to obtain global representations, thereby enabling effective interaction between global and local features. Second, unlike conventional methods that focus solely on facial expressions, our visual encoder takes raw frames as input to capture a broad range of bodily cues, thereby extending the scope of emotional signals encoded in the visual domain. Third, we employ a guided cross-modal attention mechanism in the feature-fusion stage to enhance the complementarity and synergy between audio and visual features. Finally, we develop a hybrid loss function—comprising audiovisual and unimodal (classification) losses as well as a combined divergence (metric) loss—to conduct end-to-end training. This design balances inter-modal similarity and disparity, thereby further optimizing the multimodal fusion process. Experimental results on the RAVDESS, CREMA-D, and CMU-MOSEI datasets demonstrate that AVERFormer achieves classification accuracies of 97.92%, 87.20%, and 79.40%, respectively—significantly outperforming current state-of-the-art approaches and showcasing its superior performance in audiovisual emotion recognition.