Method for Assessing the Repeatability of ChatGPT Machine Translation Results

Abstract

The article considers the repeatability of ChatGPT results over time intervals as a machine translator from Russian into English, using a method for the quantitative assessment of the weighting of each category describing the trajectory of translations of the same phrases over time. The experiment is described on a corpus of 50 phrases in Russian, translated into English weekly for 12 weeks. As a result, an array of 600 phrase pairs has been obtained each containing a phrase in Russian and its English translation. For each of the 50 source phrases and the corresponding 12 translations, a series of 12 annotations has been generated, including headings for classifying translation errors or a note about their absence. All of the translation series have been divided into six categories depending on the series trajectory: quality deterioration over the interval, quality improvement, quality variation, change in the set of errors without translation quality dynamics, translation change without translation quality dynamics, and translation series without change.