Leon Nissen, Johanna Flora Rother, Marie Heinemann, Lara Marie Reimer, Stephan Jonas, Tobias Raupach
{"title":"A randomised cross-over trial assessing the impact of AI-generated individual feedback on written online assignments for medical students.","authors":"Leon Nissen, Johanna Flora Rother, Marie Heinemann, Lara Marie Reimer, Stephan Jonas, Tobias Raupach","doi":"10.1080/0142159X.2025.2451870","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>Self-testing has been proven to significantly improve not only simple learning outcomes, but also higher-order skills such as clinical reasoning in medical students. Previous studies have shown that self-testing was especially beneficial when it was presented with feedback, which leaves the question whether an immediate and personalized feedback further encourages this effect. Therefore, we hypothesised that individual feedback has a greater effect on learning outcomes, compared to generic feedback.</p><p><strong>Materials and methods: </strong>In a randomised cross-over trial, German medical students were invited to voluntarily answer daily key-feature questions <i>via</i> an App. For half of the items they received a generalised feedback by an expert, while the feedback on the other half was generated immediately through ChatGPT. After the intervention, the students participated in a mandatory exit exam.</p><p><strong>Results: </strong>Those participants who used the app more frequently experienced a better learning outcome compared to those who did not use it frequently, even though this finding was only examined in a correlative nature. The individual ChatGPT generated feedback did not show a greater effect on exit exam scores compared to the expert comment (51.8 ± 22.0% vs. 55.8 ± 22.8%; <i>p</i> = 0.06).</p><p><strong>Conclusion: </strong>This study proves the concept of providing personalised feedback on medical questions. Despite the promising results, improved prompting and further development of the application seems necessary to strengthen the possible impact of the personalised feedback. Our study closes a research gap and holds great potential for further use not only in medicine but also in other academic fields.</p>","PeriodicalId":18643,"journal":{"name":"Medical Teacher","volume":" ","pages":"1-7"},"PeriodicalIF":3.3000,"publicationDate":"2025-01-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Teacher","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1080/0142159X.2025.2451870","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: Self-testing has been proven to significantly improve not only simple learning outcomes, but also higher-order skills such as clinical reasoning in medical students. Previous studies have shown that self-testing was especially beneficial when it was presented with feedback, which leaves the question whether an immediate and personalized feedback further encourages this effect. Therefore, we hypothesised that individual feedback has a greater effect on learning outcomes, compared to generic feedback.
Materials and methods: In a randomised cross-over trial, German medical students were invited to voluntarily answer daily key-feature questions via an App. For half of the items they received a generalised feedback by an expert, while the feedback on the other half was generated immediately through ChatGPT. After the intervention, the students participated in a mandatory exit exam.
Results: Those participants who used the app more frequently experienced a better learning outcome compared to those who did not use it frequently, even though this finding was only examined in a correlative nature. The individual ChatGPT generated feedback did not show a greater effect on exit exam scores compared to the expert comment (51.8 ± 22.0% vs. 55.8 ± 22.8%; p = 0.06).
Conclusion: This study proves the concept of providing personalised feedback on medical questions. Despite the promising results, improved prompting and further development of the application seems necessary to strengthen the possible impact of the personalised feedback. Our study closes a research gap and holds great potential for further use not only in medicine but also in other academic fields.
目的：自我测试已被证明不仅可以显著提高医学生的简单学习成果，而且可以显著提高临床推理等高阶技能。之前的研究表明，自我测试在有反馈的情况下尤其有益，这就留下了一个问题，即即时和个性化的反馈是否会进一步促进这种效果。因此，我们假设与一般反馈相比，个体反馈对学习结果有更大的影响。材料和方法：在一项随机交叉试验中，德国医科学生被邀请通过一个应用程序自愿回答每天的关键特征问题。对于一半的问题，他们得到了专家的一般性反馈，而另一半的反馈是通过ChatGPT立即生成的。干预后，学生们参加了强制性的毕业考试。结果：与那些不经常使用该应用程序的人相比，那些更频繁使用该应用程序的参与者获得了更好的学习效果，尽管这一发现只是在相关性质上进行了检验。与专家评论相比，个人ChatGPT生成的反馈对退出考试成绩的影响并不大(51.8±22.0% vs 55.8±22.8%；p = 0.06)。结论：本研究验证了针对医疗问题提供个性化反馈的概念。尽管结果令人鼓舞，但改进提示和进一步开发应用程序似乎有必要加强个性化反馈的可能影响。我们的研究填补了一个研究空白，不仅在医学上，而且在其他学术领域都有很大的应用潜力。
期刊介绍：
Medical Teacher provides accounts of new teaching methods, guidance on structuring courses and assessing achievement, and serves as a forum for communication between medical teachers and those involved in general education. In particular, the journal recognizes the problems teachers have in keeping up-to-date with the developments in educational methods that lead to more effective teaching and learning at a time when the content of the curriculum—from medical procedures to policy changes in health care provision—is also changing. The journal features reports of innovation and research in medical education, case studies, survey articles, practical guidelines, reviews of current literature and book reviews. All articles are peer reviewed.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
