Graph attention based on contextual reasoning and emotion-shift awareness for emotion recognition in conversations

Emotion recognition in conversations has recently emerged as a hot research topic owing to its increasingly important role in developing intelligent empathy services. Thoroughly exploring the conversational context and accurately capturing emotion-shift information are highly crucial for accurate emotion recognition in conversations. However, existing studies generally failed to fully understand the complex conversational context due to their insufficient capabilities in extracting and integrating emotional cues. Moreover, they mainly focused on the speaker’s emotion inertia while paying less attention to explore multi-perspective emotion-shift patterns. To address these limitations, this study proposes a novel multimodal approach, namely, GAT-CRESA (Graph ATtention based on Contextual Reasoning and Emotion-Shift Awareness). Specifically, the multi-turn global contextual reasoning module iteratively performs contextual perception and cognitive reasoning for efficiently understanding the global conversational context. Then, GAT-CRESA explores emotion-shift information among utterances from both the speaker-dependent and the global context-based perspectives. Next, the emotion-shift awareness graphs are constructed for extracting significant local-level conversational context, where edge relations are determined by the learnt emotion-shift labels. Finally, the outputs of graphs are concatenated for final emotion recognition. The loss of emotion prediction task is combined together with those of two perspective’s emotion-shift learning for guiding the training process. Experimental results show that our GAT-CRESA achieves new state-of-art records with 72.77% ACC and 72.81% wa-F1 on IEMOCAP, and 65.44% ACC and 65.04% wa-F1 on MELD, respectively. The ablation results also indicate the effectiveness and rationality of each component in our approach.