Triplet-modality group-guided incremental distillation with regularized group semantic consistency for multi-modal neural machine translation

Multi-modal Machine Translation (MMT) aims to tackle the challenge of cross-modal semantic alignment by integrating additional information from additional modalities, such as images, video, audio, and potentially other modalities. Unfortunately, collecting high-quality multi-modal data pairs is costly, leading to challenges in data scarcity or noise robustness. Most existing MMT research focuses on feature-level cross-modal fusion using these limited multi-modal data, training models from scratch without utilizing prior knowledge from established pure-text neural machine translation (NMT) models. This results in inefficient use of computational resources and cross-modal misalignment. To this end, this paper presents a triplet-modality group-guided incremental distillation approach, constrained by group-centered multi-modal semantic alignment, to extend the scope of machine translation in visual scenarios. The proposed approach preserves the translation capabilities of the pre-trained NMT model through triplet-modal group incremental distillation, while further improving translation performance through a regularized group alignment strategy, thereby enhancing machine translation ability in MMT. We conducted extensive experiments on two general-domain and two specific-domain MMT tasks. The results demonstrate that the proposed approach shows improvements over the state-of-the-art (SOTA) methods across all test sets, achieving performance gains of over 3.7%. In-depth analysis highlights the effectiveness and robustness of our method in cross-modal alignment and noisy visual scenarios. Our code is available at https://github.com/lyy-nlp/MMT_main.