Method for Assessing the Repeatability of ChatGPT Machine Translation Results

IF 0.5 Q4 COMPUTER SCIENCE, INFORMATION SYSTEMS

AUTOMATIC DOCUMENTATION AND MATHEMATICAL LINGUISTICS Pub Date : 2025-02-17 DOI:10.3103/S0005105524700365

A. Yu. Egorova, I. M. Zatsman, V. O. Romanenko

{"title":"Method for Assessing the Repeatability of ChatGPT Machine Translation Results","authors":"A. Yu. Egorova, I. M. Zatsman, V. O. Romanenko","doi":"10.3103/S0005105524700365","DOIUrl":null,"url":null,"abstract":"<p>The article considers the repeatability of ChatGPT results over time intervals as a machine translator from Russian into English, using a method for the quantitative assessment of the weighting of each category describing the trajectory of translations of the same phrases over time. The experiment is described on a corpus of 50 phrases in Russian, translated into English weekly for 12 weeks. As a result, an array of 600 phrase pairs has been obtained each containing a phrase in Russian and its English translation. For each of the 50 source phrases and the corresponding 12 translations, a series of 12 annotations has been generated, including headings for classifying translation errors or a note about their absence. All of the translation series have been divided into six categories depending on the series trajectory: quality deterioration over the interval, quality improvement, quality variation, change in the set of errors without translation quality dynamics, translation change without translation quality dynamics, and translation series without change.</p>","PeriodicalId":42995,"journal":{"name":"AUTOMATIC DOCUMENTATION AND MATHEMATICAL LINGUISTICS","volume":"58 6","pages":"453 - 460"},"PeriodicalIF":0.5000,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"AUTOMATIC DOCUMENTATION AND MATHEMATICAL LINGUISTICS","FirstCategoryId":"1085","ListUrlMain":"https://link.springer.com/article/10.3103/S0005105524700365","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

The article considers the repeatability of ChatGPT results over time intervals as a machine translator from Russian into English, using a method for the quantitative assessment of the weighting of each category describing the trajectory of translations of the same phrases over time. The experiment is described on a corpus of 50 phrases in Russian, translated into English weekly for 12 weeks. As a result, an array of 600 phrase pairs has been obtained each containing a phrase in Russian and its English translation. For each of the 50 source phrases and the corresponding 12 translations, a series of 12 annotations has been generated, including headings for classifying translation errors or a note about their absence. All of the translation series have been divided into six categories depending on the series trajectory: quality deterioration over the interval, quality improvement, quality variation, change in the set of errors without translation quality dynamics, translation change without translation quality dynamics, and translation series without change.

Abstract Image

查看原文本刊更多论文

ChatGPT机器翻译结果可重复性评估方法

本文考虑了ChatGPT结果随时间间隔的可重复性，作为从俄语到英语的机器翻译，使用了一种方法来定量评估描述同一短语随时间翻译轨迹的每个类别的权重。该实验是在一个包含50个俄语短语的语料库上描述的，每周翻译成英语，为期12周。结果，得到了一个包含600个短语对的数组，每个短语对包含一个俄语短语及其英语翻译。对于50个源短语和对应的12个翻译，每一个都生成了一系列的12个注释，包括翻译错误分类的标题或对其缺失的注释。所有的翻译系列根据其轨迹分为六类：间隔内质量恶化、质量改善、质量变化、错误集变化而没有翻译质量动态、翻译变化而没有翻译质量动态、翻译系列没有变化。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

AUTOMATIC DOCUMENTATION AND MATHEMATICAL LINGUISTICS COMPUTER SCIENCE, INFORMATION SYSTEMS-

自引率

40.00%

发文量

期刊介绍： Automatic Documentation and Mathematical Linguistics is an international peer reviewed journal that covers all aspects of automation of information processes and systems, as well as algorithms and methods for automatic language analysis. Emphasis is on the practical applications of new technologies and techniques for information analysis and processing.