Multimodal speech emotion recognition via dynamic multilevel contrastive loss under local enhancement network

Abstract

Multimodal speech emotion recognition is crucial for advancing human–computer interaction technology. Contrastive learning, due to its powerful ability of representation, is increasingly being applied to emotion recognition. Existing algorithms usually only consider samples of the same emotion as positive matching pairs, but ignore that the distances of different positive pairs are often different. For this issue, this paper designs a novel dynamic multilevel contrastive loss (DMCL), which achieves adaptive distance constraint by dynamic multilevel similarity. It generalizes positive matching pairs in different cases, assigns them different distances, and dynamically adjusts the corresponding labels while modeling. Building upon the DMCL, this paper further proposes a local enhancement attention mechanism (LEA) that enhances local information token-by-token on a global basis, which can enhance the robustness of the model to emotional mutations. By integrating the advantages of LEA and DMCL, this paper constructs an end-to-end multimodal speech emotion recognition network (LEDMCN). Finally, experimental results on the IEMOCAP and LSSED datasets validate the effectiveness of the proposed method, achieving state-of-the-art performance.