Fine-Grained Emotion Comprehension: Semisupervised Multimodal Emotion and Intensity Recognition

Abstract

The rapid advancement of deep learning and the exponential growth of multimodal data have led to increased attention on multimodal emotion analysis and comprehension in affect computing. While existing multimodal works have achieved notable results in emotion recognition, several challenges remain. First, the scarcity of public large-scale multimodal emotion datasets is attributed to the high cost of manual annotation and the subjectivity of handcrafted labels. Second, most approaches only focus on learning emotion category information, disregarding the crucial evaluation indicator of emotion intensity, which hampers the development of fine-grained emotion recognition. Third, a significant emotion semantic discrepancy exists in different modalities, and current methodologies struggle to bridge the cross-modal gap and effectively utilize a vast amount of unlabeled emotion data, hindering the production of high-quality pseudolabels and superior classification performance. To address these challenges, based on the multitask learning architecture, we propose a novel semisupervised fine-grained emotion recognition model SMEIR-net for multimodal emotion and intensity recognition. Concretely, in semisupervised learning (SSL) phase, we design multistage self-training and consistency regularization paradigm to generate high-quality pseudolabels. Then, in supervised learning phase, we leverage multimodal transformer fusion and adversarial learning to eliminate the cross-modal semantic discrepancy. Extensive experiments are conducted on three benchmark datasets, namely RAVDESS, eNTERFACE, and Lombard-GRID, to evaluate the proposed model. The series sets of experimental results demonstrate that our SSL model successfully utilizes multimodal data and available labels to transfer emotion and intensity information from labeled to unlabeled datasets. Moreover, the corresponding evaluation metrics demonstrate that the utilize high-quality pseudolabels can achieve superior emotion and intensity classification performance, which outperforms other state-of-the-art baselines under the same condition.