Den-ML: Multi-source cross-lingual transfer via denoising mutual learning

Abstract

Multi-source cross-lingual transfer aims to acquire task knowledge from multiple labelled source languages and transfer it to an unlabelled target language, enabling effective performance in this target language. The existing methods mainly focus on weighting predictions of language-specific classifiers trained in source languages to derive final results for target samples. However, we argue that, due to the language gap, language-specific classifiers inevitably generate many noisy predictions for target samples. Furthermore, these methods disregard the mutual guidance and utilization of knowledge among multiple source languages. To address these issues, we propose a novel model, Den-ML, which improves the model’s performance in multi-source scenarios through two perspectives: reducing prediction noise of language-specific classifiers and prompting mutual learning among these classifiers. Firstly, Den-ML devises a discrepancy-guided denoising learning method to learn discriminative representations for the target language, thus mitigating the noise prediction of classifiers. Secondly, Den-ML develops a pseudo-label-supervised mutual learning method, which relies on forcing probability distribution interactions among multiple language-specific classifiers for knowledge transfer, thus achieving mutual learning among classifiers. We conduct experiments on three different tasks, named entity recognition, paraphrase identification and natural language inference, with two different multi-source combination settings (same- and different-family settings) covering 39 languages. Our approach outperforms the benchmark and the SOTA model in most metrics for all three tasks in different settings. In addition, we perform ablation, visualization and analysis experiments on three different tasks, and the experimental results validate the effectiveness of our approach.