Correcting translation for non-autoregressive transformer

Abstract

Non-Autoregressive Transformer has shown great success in recent years. It generally employs the encoder–decoder framework, where the encoder maps the sentence into hidden representation, and the decoder generates the target tokens simultaneously. Since the theory of non-autoregressive transformer is consistent with the architecture of the encoder, we suppose that it is somewhat wasteful for the encoder to only map input sentence into hidden representation. In this study, we proposed a novel non-autoregressive transformer to fully exploit the capabilities of the encoder. Specifically, in our approach, the encoder not only encodes the input sentence into hidden representation, but also generates the target tokens. Consequently, the decoder is relieved of its responsibility to generate the target tokens, instead of focusing on correcting the sentence produced by the encoder. We evaluate the performance of the proposed non-autoregressive transformer on three widely-used translation tasks. The experimental results illustrate the proposed method can significantly improve the performance of the non-autoregressive transformer , which achieved 27.94 BLEU on WMT14 EN $\to$ DE task, 33.96 BLEU on WMT16 EN $\to$ RO task, and 33.85 BLEU on IWSLT14 DE $\to$ EN.