BERT-fused Model for Finnish-Swedish Translation

Abstract

Translation between Finnish and Swedish is a common yet time-consuming and expensive task. In this paper, we train new neural machine translation models and compare them with publicly available tools for automatic translation of Finnish to Swedish. Furthermore, we analyze if fusing BERT models with traditional Transformer models produces better translations. We train a base Transformer and a large Transformer model using Fairseq and compare the results with BERT-fused versions of the models. Our large transformer model matches the state-of-the-art performance in Finnish-Swedish translation and slightly improves the BLEU score from 29.4 to 29.8. In our experiments, fusing the smaller Transformer model with a pre-trained BERT improves the quality of the translations. Surprisingly, the larger Transformer model in contrast does not benefit from being fused with a BERT model.