Emotional inverse reasoning trees and dominant modality focus for emotion recognition in conversations

Abstract

Emotion recognition in conversation (ERC) is crucial to advancing human–computer interaction. However, current methods often ignore the importance of keywords in emotional expression, neglecting both the emotional information these keywords convey and their dynamic variations. In addition, previous studies have not deeply considered the characteristics and commonalities of heterogeneous modalities before fusion, leading to noise accumulation and weakened intermodal interactions. During multimodal fusion, these methods have not effectively accounted for strength differences between modalities, particularly underestimating the notable influence of the text modality on ERC. Moreover, traditional research has made only limited attempts to enhance modality representation capabilities. To address these issues, we propose the Emotional Inverse Reasoning Trees and Dominant Modal Focus model (EIRT-DMF) for ERC. The model leverages commonsense knowledge to extract keywords from utterances and introduces an innovative emotional inverse reasoning tree structure to enhance textual semantic representation and strengthen the transmission of emotional cues. Meanwhile, we design a modality optimization module to handle intra-modality associations and cross-modality interactions. In the fusion phase, the text modality is employed as the dominant modality to gain a collaborative understanding of intermodal semantics. In addition, we introduce a hybrid knowledge-distillation mechanism that employs multilevel learning to generate higher-quality multimodal representations. Experiments on the IEMOCAP and MELD datasets indicate that EIRT-DMF achieved state-of-the-art (SOTA) performance compared to all baselines.