Reproducible and generalizable speech emotion recognition via an Intelligent Fusion Network

Abstract

Speech emotion recognition (SER) is a critical aspect of enhancing the naturalness and effectiveness of Human-computer interaction systems. Despite substantial progress through deep learning techniques, challenges remain, particularly concerning model performance, reproducibility, and generalization. To address these challenges, we propose the Intelligent Fusion Network (IFN), a novel framework designed to improve emotion recognition by leveraging an isomorphic architecture and attention-based mechanisms. The IFN framework consists of five key components: an input processing layer, a feature mapping module, a dual attention mechanism, a convolutional feature refinement module, and a multiplicative fusion module, culminating in an output layer. In addition, we introduce a robust methodology for quantifying and assessing the reproducibility of deep learning models, ensuring consistent and reliable evaluations. Extensive experiments conducted across six benchmark datasets—EMODB, CASIA, SAVEE, BodEMODB, IEMOCAP, and ESD—demonstrate the superior performance of IFN. Specifically, IFN achieves an accuracy of 96.31 % on the ESD dataset, surpassing the leading baseline by 2.70 %. On the more challenging IEMOCAP dataset, IFN attains an accuracy of 64.32 %, highlighting its ability to generalize effectively across diverse datasets. Furthermore, IFN demonstrates exceptional reproducibility, with general and correct reproducibility rates of 86.69 % and 86.34 %, respectively, at k = 10 on the ESD dataset, significantly outperforming existing approaches. These results highlight IFN as a highly reliable and effective solution for advancing SER, offering the potential to enable more intuitive and efficient Human-Computer Interaction (HCI) systems.