Evaluation of the Coherence of Ukrainian Texts Using a Transformer Architecture

Abstract

The coherence of a text interprets the thematic connection between the different parts of the text. According to the lack of the unified structure of a text and the necessity to estimate the semantic meaning of its parts, different machine learning methods, namely, deep learning techniques should be used to evaluate the coherence of a text. In this paper, different baseline and state-of-the-art methods of the coherence evaluation based on deep learning paradigms have been analyzed in detail. According to the disability of analyzed methods to take into account both word ordering and long-distant relations within a text, a new Transformer-based neural network model has been proposed. The principle of work and advantages of a self-attention mechanism has been considered. The training of the proposed model has been performed on the corpus of Ukrainian news. In order to estimate the accuracy of the model and to take into account the imbalance of classes, corresponding metrics have been calculated with further detection of an appropriate threshold value of an output. The effectiveness of the model has been examined on the document discrimination and insertion tasks. The results obtained may indicate that the proposed model outperforms baseline methods and can be used to estimate the coherence of both Ukrainian texts and foreign language documents.