RMER-DT: Robust multimodal emotion recognition in conversational contexts based on diffusion and transformers

As the digital age advances, multimodal emotion recognition (MER) technology is increasingly crucial in fields like smart interaction and mental health assessment. However, emotional recognition in conversational contexts faces numerous challenges, particularly in effectively managing missing multimodal data. To address this issue, we propose RMER-DT (Robust Multimodal Emotion Recognition in Conversational Contexts based on Diffusion and Transformers), a novel MER model specifically designed for accurate emotion recognition in conversational environments while addressing the issue of random modality absence. To improve contextual dialogue-based multimodal emotion recognition, RMER-DT introduces a novel data recovery strategy and an optimized framework. By integrating diffusion models and transformer technologies, our model effectively recovers and integrates various modal data, such as audio, facial expressions, and text. Furthermore, RMER-DT enhances the semantic interaction and representation between modalities through the introduction of positional embeddings and speaker embeddings. Experimental results on the MELD and IEMOCAP datasets demonstrate significant advantages of RMER-DT over existing technologies in handling MER tasks and enhancing the accuracy and robustness of emotion recognition.