Multimodal Sentiment and Emotion Detection Using Hybrid Fusion of Text, Audio, and Image with Explainable AI Integration

Abstract

Emotion recognition plays a crucial role in applications such as human-computer interaction, healthcare, adaptive learning, and social media analysis. However, existing multimodal frameworks often struggle to effectively model paralinguistic features and manage the random fusion of different modalities, resulting in misclassifications and limited interpretability. To address these challenges, this paper introduces TriModal-SentiXNet, a novel multimodal framework designed for dual emotion and sentiment recognition using text, speech, and facial expressions. The framework operates in three main stages: pre-processing, feature extraction, and explainable classification. The pre-processing phase employs a Multimodal Adaptive Pre-processing (MAP) method that enhances input quality by denoising speech, standardizing visual frames, and semantically enriching textual data. For feature extraction, the RoBi-FuseNet architecture learns comprehensive contextual representations from each modality while effectively modeling temporal dynamics and cross-modal interactions to improve recognition accuracy. Explainable classification is achieved using SHapley Additive exPlanations (SHAP), which provides transparency by quantifying each modality’s contribution and ensuring balanced influence. The integration of adaptive pre-processing and context-aware fusion strengthens robustness, interpretability, and overall system reliability. Experimental evaluations on benchmark datasets CMU-MOSEI and CMU-MOSI demonstrate notable performance gains, achieving accuracies of 87.92 and 86%, respectively. These results surpass unimodal and existing multimodal baselines, establishing TriModal-SentiXNet as a promising step toward ethical, transparent, and context-aware emotion recognition systems suitable for real-world deployment.