Assessing student perceptions and use of instructor versus AI-generated feedback

IF 6.7 1区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

British Journal of Educational Technology Pub Date : 2024-12-27 DOI:10.1111/bjet.13558

Erkan Er, Gökhan Akçapınar, Alper Bayazıt, Omid Noroozi, Seyyed Kazem Banihashem

{"title":"Assessing student perceptions and use of instructor versus AI-generated feedback","authors":"Erkan Er, Gökhan Akçapınar, Alper Bayazıt, Omid Noroozi, Seyyed Kazem Banihashem","doi":"10.1111/bjet.13558","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <p>Despite the growing research interest in the use of large language models for feedback provision, it still remains unknown how students perceive and use AI-generated feedback compared to instructor feedback in authentic settings. To address this gap, this study compared instructor and AI-generated feedback in a Java programming course through an experimental research design where students were randomly assigned to either condition. Both feedback providers used the same assessment rubric, and students were asked to improve their work based on the feedback. The feedback perceptions scale and students' laboratory assignment scores were compared in both conditions. Results showed that students perceived instructor feedback as significantly more useful than AI feedback. While instructor feedback was also perceived as more fair, developmental and encouraging, these differences were not statistically significant. Importantly, students receiving instructor feedback showed significantly greater improvements in their lab scores compared to those receiving AI feedback, even after controlling for their initial knowledge levels. Based on the findings, we posit that AI models potentially need to be trained on data specific to educational contexts and hybrid feedback models that combine AI's and instructors' strengths should be considered for effective feedback practices.</p>\n </section>\n \n <section>\n \n <div>\n \n <div>\n \n <h3>Practitioner notes</h3>\n <p>What is already known about this topic\n </p><ul>\n \n <li>Feedback is crucial for student learning in programming education.</li>\n \n <li>Providing detailed personalised feedback is challenging for instructors.</li>\n \n <li>AI-powered solutions like ChatGPT can be effective in feedback provision.</li>\n \n <li>Existing research is limited and shows mixed results about AI-generated feedback.</li>\n </ul>\n \n <p>What this paper adds\n </p><ul>\n \n <li>The effectiveness of AI-generated feedback was compared to instructor feedback.</li>\n \n <li>Both feedback types received positive perceptions, but instructor feedback was seen as more useful.</li>\n \n <li>Instructor feedback led to greater score improvements in the programming task.</li>\n </ul>\n \n <p>Implications for practice and/or policy\n </p><ul>\n \n <li>AI should not be the sole source of feedback, as human expertise is crucial.</li>\n \n <li>AI models should be trained on context-specific data to improve feedback actionability.</li>\n \n <li>Hybrid feedback models should be considered for a scalable and effective approach.</li>\n </ul>\n \n </div>\n </div>\n </section>\n </div>","PeriodicalId":48315,"journal":{"name":"British Journal of Educational Technology","volume":"56 3","pages":"1074-1091"},"PeriodicalIF":6.7000,"publicationDate":"2024-12-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"British Journal of Educational Technology","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/bjet.13558","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Despite the growing research interest in the use of large language models for feedback provision, it still remains unknown how students perceive and use AI-generated feedback compared to instructor feedback in authentic settings. To address this gap, this study compared instructor and AI-generated feedback in a Java programming course through an experimental research design where students were randomly assigned to either condition. Both feedback providers used the same assessment rubric, and students were asked to improve their work based on the feedback. The feedback perceptions scale and students' laboratory assignment scores were compared in both conditions. Results showed that students perceived instructor feedback as significantly more useful than AI feedback. While instructor feedback was also perceived as more fair, developmental and encouraging, these differences were not statistically significant. Importantly, students receiving instructor feedback showed significantly greater improvements in their lab scores compared to those receiving AI feedback, even after controlling for their initial knowledge levels. Based on the findings, we posit that AI models potentially need to be trained on data specific to educational contexts and hybrid feedback models that combine AI's and instructors' strengths should be considered for effective feedback practices.

Practitioner notes

What is already known about this topic

Feedback is crucial for student learning in programming education.
Providing detailed personalised feedback is challenging for instructors.
AI-powered solutions like ChatGPT can be effective in feedback provision.
Existing research is limited and shows mixed results about AI-generated feedback.

What this paper adds

The effectiveness of AI-generated feedback was compared to instructor feedback.
Both feedback types received positive perceptions, but instructor feedback was seen as more useful.
Instructor feedback led to greater score improvements in the programming task.

Implications for practice and/or policy

AI should not be the sole source of feedback, as human expertise is crucial.
AI models should be trained on context-specific data to improve feedback actionability.
Hybrid feedback models should be considered for a scalable and effective approach.

查看原文本刊更多论文

评估学生的看法和使用教师与人工智能生成的反馈

尽管对使用大型语言模型提供反馈的研究兴趣日益浓厚，但与真实环境中的教师反馈相比，学生如何感知和使用人工智能生成的反馈仍然未知。为了解决这一差距，本研究通过实验研究设计比较了Java编程课程中教师和人工智能生成的反馈，学生被随机分配到任何一种情况。两个反馈提供者使用相同的评估标准，学生被要求根据反馈改进他们的工作。在两种情况下，比较反馈感知量表和学生实验作业得分。结果显示，学生认为教师的反馈比人工智能的反馈更有用。虽然教师的反馈也被认为更公平，发展和鼓励，但这些差异在统计上并不显著。重要的是，与接受人工智能反馈的学生相比，接受教师反馈的学生在实验成绩上的进步明显更大，即使在控制了他们的初始知识水平之后也是如此。基于这些发现，我们认为人工智能模型可能需要接受特定于教育背景的数据训练，并且应该考虑将人工智能和教师的优势结合起来的混合反馈模型，以进行有效的反馈实践。关于这个话题，我们已经知道反馈对于学生在编程教育中的学习是至关重要的。对教师来说，提供详细的个性化反馈是一项挑战。ChatGPT等人工智能解决方案可以有效地提供反馈。现有的研究是有限的，并且对人工智能产生的反馈显示出不同的结果。将人工智能生成的反馈与教师反馈的有效性进行了比较。两种反馈都得到了积极的评价，但教师的反馈被认为更有用。教师的反馈使得编程任务的分数有了更大的提高。人工智能对实践和/或政策的影响不应成为反馈的唯一来源，因为人类的专业知识至关重要。人工智能模型应该在特定情境的数据上进行训练，以提高反馈的可操作性。混合反馈模型是一种可扩展和有效的方法。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

British Journal of Educational Technology EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

15.60

自引率

4.50%

发文量

111

期刊介绍： BJET is a primary source for academics and professionals in the fields of digital educational and training technology throughout the world. The Journal is published by Wiley on behalf of The British Educational Research Association (BERA). It publishes theoretical perspectives, methodological developments and high quality empirical research that demonstrate whether and how applications of instructional/educational technology systems, networks, tools and resources lead to improvements in formal and non-formal education at all levels, from early years through to higher, technical and vocational education, professional development and corporate training.