Exploring the Effectiveness of Employing Limited Resources for Deep Neural Pairwise Evaluation of Machine Translation

Abstract

In this paper, a light resource learning schema, i.e. a schema that depends on limited resources, is introduced, which aims to choose the better translation between two machine translation (MT) outputs, based on information regarding the source segments (SSE) and string-based features. A concatenation of vectors, including mathematically calculated embeddings from the SSE, statistical MT (SMT) and neural MT (NMT) segments (S1 and S2 respectively), are used as input to a neural network (NN). Experiments are run for two different forms of text structure (one that is a formal, well-structured corpus (C2) and one that is informal (C1)) for the English (EN) – Greek (EL) language pair. Instead of relying on high-level experts’ annotations, a novel automatic metric is proposed for determining the better translation, namely the quality estimation (QE) score. This score is based on string-based features derived from both the SSE and the MT segments. Experimental results demonstrate a quite good performance for the proposed feed-forward NN, comparable to the existing state of-the-art models for MT evaluation that require more sophisticated resources.