Comparison of AI-Generated and Instructor Feedback: No Significant Difference in Perceived Feedback Quality and Neither on Performance

IF 4.6 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Computer Assisted Learning Pub Date : 2025-09-22 DOI:10.1111/jcal.70134

Şeyma Çağlar-Özhan, Perihan Tekeli, Selay Arkün-Kocadere

{"title":"Comparison of AI-Generated and Instructor Feedback: No Significant Difference in Perceived Feedback Quality and Neither on Performance","authors":"Şeyma Çağlar-Özhan, Perihan Tekeli, Selay Arkün-Kocadere","doi":"10.1111/jcal.70134","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Feedback is an essential part of the educational process as it enriches students' learning experiences, provides information about their current performance, shows them what is lacking in achieving goals, and provides guidance on the strategies needed to achieve those goals. Teachers, especially in crowded classrooms, often have difficulty allocating enough time to provide feedback. ChatGPT has the potential to deliver feedback more efficiently and effectively.</p>\n </section>\n \n <section>\n \n <h3> Objective</h3>\n \n <p>This study aims to investigate the effectiveness of ChatGPT-generated feedback in students' project proposal development process. Therefore, ChatGPT-generated and instructor's feedback are compared in terms of perceived feedback quality and performance of students.</p>\n </section>\n \n <section>\n \n <h3> Method</h3>\n \n <p>In the study, a post-test control group experimental design was used. A total of 48 students enrolled in a blended course at a state university participated in the study for 6 weeks. The study was conducted using a randomised post-test-only control group experimental design. Students were asked to prepare a project proposal and feedback was given on each task. While the control group received instructor feedback, the experimental group received ChatGPT feedback. Data were collected via the Formative Feedback Perceptions Scale, and the students' project proposals were graded based on the rubric to evaluate their performance.</p>\n </section>\n \n <section>\n \n <h3> Results and Conclusion</h3>\n \n <p>According to the Mann–Whitney <i>U</i> test analysis, there was no statistically significant difference between the experimental and the control group regarding both students' perceived feedback quality and performance. In conclusion, it can be argued that when guided by the appropriate prompts and asked to examine assignments of similar complexity to those in this study, ChatGPT can provide feedback of similar quality and equivalence to the feedback provided by instructors. In this way, it seems possible to improve students' learning as much as an instructor with individualised feedback by ChatGPT. Finally, it should be noted that these findings are within the limitations of the specific context of project proposal development, reliance on self-reported data, and the experimental nature of the study; it is recommended to evaluate the effectiveness of AI-generated feedback in different contexts.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"41 5","pages":""},"PeriodicalIF":4.6000,"publicationDate":"2025-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.70134","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Feedback is an essential part of the educational process as it enriches students' learning experiences, provides information about their current performance, shows them what is lacking in achieving goals, and provides guidance on the strategies needed to achieve those goals. Teachers, especially in crowded classrooms, often have difficulty allocating enough time to provide feedback. ChatGPT has the potential to deliver feedback more efficiently and effectively.

Objective

This study aims to investigate the effectiveness of ChatGPT-generated feedback in students' project proposal development process. Therefore, ChatGPT-generated and instructor's feedback are compared in terms of perceived feedback quality and performance of students.

Method

In the study, a post-test control group experimental design was used. A total of 48 students enrolled in a blended course at a state university participated in the study for 6 weeks. The study was conducted using a randomised post-test-only control group experimental design. Students were asked to prepare a project proposal and feedback was given on each task. While the control group received instructor feedback, the experimental group received ChatGPT feedback. Data were collected via the Formative Feedback Perceptions Scale, and the students' project proposals were graded based on the rubric to evaluate their performance.

Results and Conclusion

According to the Mann–Whitney U test analysis, there was no statistically significant difference between the experimental and the control group regarding both students' perceived feedback quality and performance. In conclusion, it can be argued that when guided by the appropriate prompts and asked to examine assignments of similar complexity to those in this study, ChatGPT can provide feedback of similar quality and equivalence to the feedback provided by instructors. In this way, it seems possible to improve students' learning as much as an instructor with individualised feedback by ChatGPT. Finally, it should be noted that these findings are within the limitations of the specific context of project proposal development, reliance on self-reported data, and the experimental nature of the study; it is recommended to evaluate the effectiveness of AI-generated feedback in different contexts.

查看原文本刊更多论文

人工智能生成的反馈与教师反馈的比较：在感知反馈质量上无显著差异，在表现上也无显著差异

反馈是教育过程中必不可少的一部分，因为它丰富了学生的学习经验，提供了关于他们当前表现的信息，告诉他们在实现目标方面缺乏什么，并为实现这些目标所需的策略提供指导。教师，尤其是在拥挤的教室里，往往很难分配足够的时间来提供反馈。ChatGPT具有更有效地交付反馈的潜力。目的本研究旨在探讨chatgpt生成的反馈在学生项目提案开发过程中的有效性。因此，将chatgpt生成的反馈和教师的反馈在学生感知反馈质量和表现方面进行比较。方法采用测后对照组试验设计。一所州立大学共有48名学生参加了为期6周的混合课程研究。本研究采用随机后验对照组实验设计。学生们被要求准备一份项目建议书，并就每项任务给出反馈。对照组接受教练反馈，实验组接受ChatGPT反馈。数据通过形成反馈感知量表收集，学生的项目建议根据标题进行评分，以评估他们的表现。结果与结论根据Mann-Whitney U检验分析，实验组与对照组在学生感知反馈质量和表现上均无统计学差异。总之，可以认为，当被适当的提示引导并被要求检查与本研究中相似复杂性的作业时，ChatGPT可以提供与教师提供的反馈相似的质量和等效性的反馈。这样，通过ChatGPT的个性化反馈，似乎可以像教师一样提高学生的学习效果。最后，应该指出的是，这些发现是在项目提案制定的具体背景、对自我报告数据的依赖以及研究的实验性质的限制之内的；建议在不同情况下评估人工智能生成的反馈的有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Computer Assisted Learning EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.70

自引率

6.00%

发文量

116

期刊介绍： The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope