Can generative AI figure out figurative language? The influence of idioms on essay scoring by ChatGPT, Gemini, and Deepseek

IF 5.5 1区文学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Assessing Writing Pub Date : 2025-09-06 DOI:10.1016/j.asw.2025.100981

Enis Oğuz

{"title":"Can generative AI figure out figurative language? The influence of idioms on essay scoring by ChatGPT, Gemini, and Deepseek","authors":"Enis Oğuz","doi":"10.1016/j.asw.2025.100981","DOIUrl":null,"url":null,"abstract":"<div><div>The developments in Generative AI technologies have paved the way for numerous innovations in different fields. Recently, Generative AI has been proposed as a competitor to AES systems in evaluating student essays. Considering the potential limitations of AI in processing idioms, this study assessed the scoring performances of Generative AI models for essays with and without idioms by incorporating insights from Corpus Linguistics and Computational Linguistics. Two equal essay lists were created from 348 student essays taken from a corpus: one with multiple idioms present in each essay and another with no idioms. Three Generative AI models (ChatGPT, Gemini, and Deepseek) were asked to score all essays in both lists three times, using the same rubric used by human raters. The results revealed excellent consistency for all models without any detectable bias for any demographic group, but Gemini outperformed its competitors in interrater reliability with human raters. For essays with idioms, Gemini followed the most similar pattern to human raters. While models in the study demonstrated capability for a hybrid approach in formative assessments, Gemini was the best candidate due to its ability to handle figurative language and its potential for handling essay-scoring tasks in high-stakes exams in the future.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100981"},"PeriodicalIF":5.5000,"publicationDate":"2025-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Assessing Writing","FirstCategoryId":"98","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1075293525000686","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

The developments in Generative AI technologies have paved the way for numerous innovations in different fields. Recently, Generative AI has been proposed as a competitor to AES systems in evaluating student essays. Considering the potential limitations of AI in processing idioms, this study assessed the scoring performances of Generative AI models for essays with and without idioms by incorporating insights from Corpus Linguistics and Computational Linguistics. Two equal essay lists were created from 348 student essays taken from a corpus: one with multiple idioms present in each essay and another with no idioms. Three Generative AI models (ChatGPT, Gemini, and Deepseek) were asked to score all essays in both lists three times, using the same rubric used by human raters. The results revealed excellent consistency for all models without any detectable bias for any demographic group, but Gemini outperformed its competitors in interrater reliability with human raters. For essays with idioms, Gemini followed the most similar pattern to human raters. While models in the study demonstrated capability for a hybrid approach in formative assessments, Gemini was the best candidate due to its ability to handle figurative language and its potential for handling essay-scoring tasks in high-stakes exams in the future.

查看原文本刊更多论文

生成式人工智能能理解比喻语言吗？习语对ChatGPT、Gemini和Deepseek作文评分的影响

生成式人工智能技术的发展为不同领域的众多创新铺平了道路。最近，生成人工智能被提议作为AES系统在评估学生论文方面的竞争对手。考虑到人工智能在处理成语方面的潜在局限性，本研究通过结合语料库语言学和计算语言学的见解，评估了生成式人工智能模型在有成语和没有成语的文章中的评分表现。从一个语料库中提取的348篇学生论文创建了两个相等的论文列表：一篇文章中有多个习语，另一篇没有习语。三个生成式人工智能模型（ChatGPT、Gemini和Deepseek）被要求对两个列表中的所有文章进行三次评分，使用与人类评分者相同的评分标准。结果显示，所有模型都具有出色的一致性，对任何人口统计群体都没有任何可检测到的偏差，但Gemini在与人类评分者的相互可靠性方面优于其竞争对手。对于带有习语的作文，双子座的评分模式与人类最相似。虽然研究中的模型显示了在形成性评估中采用混合方法的能力，但双子座是最佳人选，因为它有能力处理比喻性语言，并且有潜力在未来的高风险考试中处理论文评分任务。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Assessing Writing Multiple-

CiteScore

6.00

自引率

17.90%

发文量

期刊介绍： Assessing Writing is a refereed international journal providing a forum for ideas, research and practice on the assessment of written language. Assessing Writing publishes articles, book reviews, conference reports, and academic exchanges concerning writing assessments of all kinds, including traditional (direct and standardised forms of) testing of writing, alternative performance assessments (such as portfolios), workplace sampling and classroom assessment. The journal focuses on all stages of the writing assessment process, including needs evaluation, assessment creation, implementation, and validation, and test development.