Improving speech emotion recognition using gated cross-modal attention and multimodal homogeneous feature discrepancy learning

Speech emotion recognition (SER) remains a significant and crucial challenge due to the complex and multifaceted nature of human emotions. To tackle this challenge, researchers strive to integrate information from diverse modalities through multimodal learning. However, existing multimodal fusion techniques often overlook the intricacies of interactions between different modalities, resulting in suboptimal feature representations. In this paper, we propose WavFusion, a multimodal framework designed for SER that tackles key research challenges, such as effective multimodal fusion, modality heterogeneity, and discriminative representation learning. By utilizing a gated cross-modal attention mechanism and multimodal homogeneous feature discrepancy learning, WavFusion outperforms existing state-of-the-art methods on benchmark datasets. Our research highlights the importance of capturing subtle cross-modal interactions and learning discriminative representations for accurate multimodal SER. Experimental results indicate that the proposed method is highly competitive and better than most of the latest state-of-the-art methods for SER. WavFusion achieves 0.78 % and 1.27 % improvement in accuracy and 0.74 % and 0.44 % improvement in weighted F1 score over the previous methods on the IEMOCAP and MELD datasets, respectively.