Multimodal disentanglement implicit distillation for speech emotion recognition

Audio signals are generally utilized with textual data for speech emotion recognition. Nevertheless, cross-modal interactions suffer from distribution discrepancy and information redundancy, leading to an inaccurate multimodal representation. Hence, this paper proposes a multimodal disentanglement implicit distillation model (MDID) that excavates and exploits each modality’s sentiment and specific characteristics. Specifically, the pre-trained models extract high-level acoustic and textual features and align them via an attention mechanism. Then, each modality is disentangled into modality sentiment-specific features. Subsequently, feature-level and logit-level distillation distill the purified modality-specific feature into the modality-sentiment feature. Compared to the adaptive fusion feature, solely employing the refined modality-sentiment feature yields superior performance for emotion recognition. Comprehensive experiments on the IEMOCAP and RAVDESS datasets indicate that MDID outperforms state-of-the-art approaches.