Speech emotion recognition algorithm based on bimodality and attention mechanism

To address the problem of low accuracy of unimodal speech emotion recognition methods, a bimodal MCNN-BiLSTM-Attention speech emotion recognition algorithm is proposed. The algorithm adopts the Mel-spectrogram and text information in audio as input, constructs a bimodal algorithm with attention mechanism based on convolutional neural network CNN and bi-directional long and short-term memory network BiLSTM, respectively, and uses Early fusion, Feature fusion and data augmentation to improve the classification accuracy. The algorithm achieves WA and UA accuracies of 74.10% and 77.10% on the IEMOCAP dataset and 59.90% and 52.80% on the MELD dataset, respectively, which are significantly improved compared with the single-modal approach.