Emotional conflict adaptation for multimodal sentiment analysis

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer user sentiment by integrating textual, auditory, and visual cues, providing richer and more nuanced emotional understanding than unimodal approaches. However, existing models often struggle to distinguish emotionally consistent from conflicting cases, limiting their effectiveness in complex scenarios. Empirical evidence shows that most multimodal samples exhibit inter-modal emotional consistency, while a smaller portion involves emotional conflict. To address this, we propose Emotional Conflict Adaptation (ECA), a framework designed to handle both cases through tailored strategies. ECA comprises a Unimodal Network that extracts modality-specific representations and quantifies emotional conflict intensity, and a dual-branch Multimodal Fusion Network that adaptively integrates predictions from easy and hard branches based on the quantified intensity. The hard branch further incorporates Decouple Attention and intra-/inter-modal contrastive learning to disentangle features and enhance conflict resolution. Extensive experiments on CH-SIMS and CMU-MOSEI demonstrate that ECA achieves state-of-the-art performance, particularly in emotionally complex conditions, underscoring the importance of explicitly modeling emotional consistency and conflict in MSA.