Multi-task coordinate attention gating network for speech emotion recognition under noisy circumstances

Abstract

Speech emotion recognition (SER) has recently made great progress in ideal environments, but their performance deteriorates dramatically when applied in complex real-world environments, mainly due to poor model robustness and generalization ability. To this end, we propose a multi-task coordinate attention gated network (MTCAGN) framework. For the SER main task, we propose a multi-scale gated convolutional neural network model with the coordinate attention mechanism, which captures a wide range of emotional features at different scales and the key global information, accurately focusing on salient emotional features in speech signals. Speech enhancement is used as an auxiliary task during the training phase, and the overall robustness of the system is strengthened through shared representation learning, allowing it to withstand complex interferences in noisy scenarios. In the inference phase, the speech enhancement branch is removed and only the SER task is retained. Therefore, our proposed method improves the robustness of the SER system without increasing inference complexity. To simulate the noise scenario, we construct three noisy speech datasets by randomly mixing clean audio from IEMOCAP or EMODB dataset with noise from the MUSAN dataset. The empirical findings evince that our proposed model exhibits superior performance in challenging low signal-to-noise ratio environments compared to the present state-of-the-art techniques, as indicated by weighted and unweighted accuracy metrics.