An adaptation framework with unified embedding reconstruction for cross-corpus speech emotion recognition

Abstract

It is challenging for speech emotion recognition (SER) to maintain robustness under cross-domain scenarios. Unsupervised domain adaptation (UDA) algorithms have been explored to address the domain shift in SER without relying on emotion labels in the target domain. As a promising framework in UDAs, self-supervised learning (SSL)-based domain exploration (SDE) investigates the domain and structural information within the target domain, aligning domain discrepancies while preserving the model’s emotion discrimination capability. However, SSL often inadvertently introduces emotion-irrelevant information, adversely affecting the UDA performance. To resolve this, we introduce a novel UDA framework called unified SDE (U-SDE), where both source and target domains conduct a unified SSL task. In the source domain, U-SDE guides the source SSL to focus on emotion-related information due to supervised emotion classification constraints. Simultaneously, in the target domain, shared network weights enable the target SSL branch to concentrate on intrinsic emotional and domain features. However, simply using existing SSL algorithms to implement this framework might disrupt the training of the supervised SER branch. To overcome this, we propose the embedding reconstruction of masked speech (ERMS) algorithm. In ERMS, the emotion encoder transforms the embedding of the masked speech to match the embedding of its corresponding unmasked speech, thereby capturing the emotion discriminative feature within the sample. Finally, we employ ERMS to realize the proposed U-SDE paradigm, termed unified ERMS (U-ERMS). We conducted systematic cross-domain SER experiments by designing 52 scenarios using seven well-known datasets. Experimental results showed that the proposed U-ERMS achieved state-of-the-art performance in cross-domain SERs.