EmoVerse: Enhancing Multimodal Large Language Models for Affective Computing via Multitask Learning

Abstract

Affective computing is essential for applications such as human–computer interaction. While Multimodal Large Language Models (MLLMs) demonstrate impressive general capabilities, they face considerable challenges in affective computing, particularly in detecting subtle facial expressions and handling complex affective tasks, such as emotion reason inference and understanding emotions in multimodal long-context scenarios. Furthermore, there is a lack of a unified MLLM that can effectively handle a wide range of affective tasks. To address these challenges, we propose Emotion Universe (EmoVerse), a MLLM trained through a Multistage Multitask Sentiment and Emotion (M2SE) instruction tuning strategy. Through this training strategy, EmoVerse acquires the ability to deeply analyze the underlying reasons for affective states. Besides, to address the lack of multitask datasets in affective computing, we introduce the Affective Multitask (AMT) Dataset, which supports multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction tasks. Extensive experiments demonstrate that EmoVerse outperforms existing methods, achieving state-of-the-art results in affective tasks. The code is available at https://github.com/liaolea/EmoVerse.