Modality-Guided Refinement Learning for Multimodal Emotion Recognition

Abstract

Multimodal emotion recognition (MER) aims to understand human emotions by leveraging multiple modalities. Previous MER methods have focused on learning enhanced multimodal representations through various interaction and fusion mechanisms, utilizing different types of features from individual modalities. However, these methods often fail to account for the varying contributions of each modality to emotion, leading to suboptimal representations. To address this, we propose a modality-guided refinement learning framework that enhances multimodal representations by incorporating modality information. Specifically, we decouple multimodal representations into modality-invariant and modality-specific components by introducing shared and private encoders, which are learned by leveraging the distributional properties of the representations in their latent subspaces, guided by a modality classifier. Our method introduces margin constraints to further refine these decoupled representations, adaptively considering the contribution of each modality during the decoupling and multimodal learning processes. This optimization reduces information loss and corruption, resulting in more robust and discriminative multimodal representation learning. We evaluate our proposed method through experiments on two benchmark MER datasets: the CMU Multimodal Corpus of Sentiment Intensity (CMU-MOSI) and the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI). Comprehensive experiments demonstrate that our method outperforms several baseline models in multimodal emotion recognition.