Evaluating the evaluators: a comparative study of AI and Teacher Assessments in Higher Education

IF 1.2 Q2 EDUCATION & EDUCATIONAL RESEARCH

Digital Education Review Pub Date : 2024-07-01 DOI:10.1344/der.2024.45.124-140

Tugra Karademir coskun, Ayfer Alper

{"title":"Evaluating the evaluators: a comparative study of AI and Teacher Assessments in Higher Education","authors":"Tugra Karademir coskun, Ayfer Alper","doi":"10.1344/der.2024.45.124-140","DOIUrl":null,"url":null,"abstract":"This study aims to examine the potential differences between teacher evaluations and artificial intelligence (AI) tool-based assessment systems in university examinations. The research has evaluated a wide spectrum of exams including numerical and verbal course exams, exams with different assessment styles (project, test exam, traditional exam), and both theoretical and practical course exams. These exams were selected using a criterion sampling method and were analyzed using Bland-Altman Analysis and Intraclass Correlation Coefficient (ICC) analyses to assess how AI and teacher evaluations performed across a broad range. The research findings indicate that while there is a high level of proficiency between the total exam scores assessed by artificial intelligence and teacher evaluations; medium consistency was found in the evaluation of visually-based exams, low consistency in video exams, high consistency in test exams, and low consistency in traditional exams. This research is crucial as it helps to identify specific areas where artificial intelligence can either complement or needs improvement in educational assessment, guiding the development of more accurate and fair evaluation tools.","PeriodicalId":44576,"journal":{"name":"Digital Education Review","volume":null,"pages":null},"PeriodicalIF":1.2000,"publicationDate":"2024-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Digital Education Review","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1344/der.2024.45.124-140","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

This study aims to examine the potential differences between teacher evaluations and artificial intelligence (AI) tool-based assessment systems in university examinations. The research has evaluated a wide spectrum of exams including numerical and verbal course exams, exams with different assessment styles (project, test exam, traditional exam), and both theoretical and practical course exams. These exams were selected using a criterion sampling method and were analyzed using Bland-Altman Analysis and Intraclass Correlation Coefficient (ICC) analyses to assess how AI and teacher evaluations performed across a broad range. The research findings indicate that while there is a high level of proficiency between the total exam scores assessed by artificial intelligence and teacher evaluations; medium consistency was found in the evaluation of visually-based exams, low consistency in video exams, high consistency in test exams, and low consistency in traditional exams. This research is crucial as it helps to identify specific areas where artificial intelligence can either complement or needs improvement in educational assessment, guiding the development of more accurate and fair evaluation tools.

查看原文本刊更多论文

评估评估者：高等教育中人工智能和教师评估的比较研究

本研究旨在探讨大学考试中教师评价与基于人工智能（AI）工具的评估系统之间的潜在差异。研究评估了各种考试，包括数字和语言课程考试、不同评估风格的考试（项目、测试考试、传统考试）以及理论和实践课程考试。这些考试采用标准抽样法进行选择，并使用布兰德-阿尔特曼分析法和类内相关系数（ICC）分析法进行分析，以评估人工智能和教师评价在广泛范围内的表现。研究结果表明，虽然人工智能评估的考试总分与教师评价之间的熟练程度很高；但在基于视觉的考试评价中发现了中等一致性，在视频考试中发现了低一致性，在测试考试中发现了高一致性，而在传统考试中发现了低一致性。这项研究至关重要，因为它有助于确定人工智能在教育评价中可以补充或需要改进的具体领域，从而指导开发更准确、更公平的评价工具。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Digital Education Review EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

3.50

自引率

14.30%

发文量

审稿时长

15 weeks

期刊介绍： Digital Education Review (DER) is a scientific, open and peer review journal designed as a space for dialogue and reflection about the impact of ICT on education and new emergent forms of teaching and learning in digital environments. It is published half-yearly (June & December) and it includes articles in English or Spanish. ICT plays an important role in education, raising discussions and important new challenges. Analyze the impact of ICT, new forms of literacy and virtual teaching and learning are the main goals of Digital Education Review. The publication is open to all those investigators who wish to propose articles on this subject. Articles admitted include empirical investigations as well as reviews and theoretical reflections. The journal publishes different kinds of articles: Peer Review Articles: articles that have passed the blind review carried out by a group of experts Reviews: short articles about books, software or websides and PhD Guest and Invited Articles: articles approved by the Editorial Board of the journal. DER publishes issues related with its focus and scope and also monographic issues, centered on a specific subject. Both of them are subjected to a peer review process. Finally, this journal is published by the Digital Education Observatory (OED) and Virtual Teaching and Learning Research Group (GREAV) at the Universitat de Barcelona.