Quasi Character-Level Transformers to Improve Neural Machine Translation on Small Datasets

Abstract

In the Neural Machine Translation community, it is a common practice to use some form of subword segmentation to encode words as a sequence of subword units. This allows practitioners to represent their entire dataset using the least amount of tokens, thus avoiding memory and performance-related problems derived from the full wordor purely character-level representations. Even though there is strong evidence that each dataset has an optimal vocabulary size, in practice it is common to use as many “words” as possible. In this work, we show how this standard approach might be counter-productive for small datasets or low-resource environments, where models trained with quasi character-level vocabularies seem to con-sistently outperform models with large subword vocabularies. Nonetheless, these improvements come at the expense of requiring a neural architecture capable of dealing with long sequences and long-term dependencies.