教育者能否区分医科学生和生成式人工智能撰写的反思?

IF 5.2 1区 教育学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES
Constance Wraith, Alasdair Carnegy, Celia Brown, Ana Baptista, Amir H Sam
{"title":"教育者能否区分医科学生和生成式人工智能撰写的反思?","authors":"Constance Wraith, Alasdair Carnegy, Celia Brown, Ana Baptista, Amir H Sam","doi":"10.1111/medu.15750","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>Reflection is integral to the modern doctor's practice and, whilst it can take many forms, written reflection is commonly found on medical school curricula. Generative artificial intelligence (GenAI) is increasingly being used, including in the completion of written assignments in medical curricula. We sought to explore if educators can distinguish between GenAI- and student-authored reflections and what features they use to do so.</p><p><strong>Methods: </strong>This was a mixed-methods study. Twenty-eight educators attended a 'think aloud' interview and were presented with a set of four reflections, either all authored by students, all by GenAI or a mixture. They were asked to identify who they thought had written the reflection, speaking aloud whilst they did so. Sensitivity (AI reflections correctly identified) and specificity (student reflections correctly identified) were then calculated, and the interview transcripts were analysed using thematic analysis.</p><p><strong>Results: </strong>Educators were unable to reliably distinguish between student and GenAI-authored reflections. Sensitivity across the four reflections ranged from 0.36 (95% CI: 0.16-0.61) to 0.64 (95% CI: 0.39-0.84). Specificity ranged from 0.64 (95% CI: 0.39-0.84) to 0.86 (95% CI: 0.60-0.96). Thematic analysis revealed three main themes when considering what features of the reflection educators used to make judgements about authorship: features of writing, features of reflection and educators' preconceptions and experiences.</p><p><strong>Discussion: </strong>This study demonstrates the challenges in differentiating between student- and GenAI-authored reflections, as well as highlighting the range of factors that influence this decision. Rather than developing ways to more accurately make this distinction or trying to stop students using GenAI, we suggest it could instead be harnessed to teach students reflective practice skills, and help students for whom written reflection in particular may be challenging.</p>","PeriodicalId":18370,"journal":{"name":"Medical Education","volume":" ","pages":""},"PeriodicalIF":5.2000,"publicationDate":"2025-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Can educators distinguish between medical student and generative AI-authored reflections?\",\"authors\":\"Constance Wraith, Alasdair Carnegy, Celia Brown, Ana Baptista, Amir H Sam\",\"doi\":\"10.1111/medu.15750\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Introduction: </strong>Reflection is integral to the modern doctor's practice and, whilst it can take many forms, written reflection is commonly found on medical school curricula. Generative artificial intelligence (GenAI) is increasingly being used, including in the completion of written assignments in medical curricula. We sought to explore if educators can distinguish between GenAI- and student-authored reflections and what features they use to do so.</p><p><strong>Methods: </strong>This was a mixed-methods study. Twenty-eight educators attended a 'think aloud' interview and were presented with a set of four reflections, either all authored by students, all by GenAI or a mixture. They were asked to identify who they thought had written the reflection, speaking aloud whilst they did so. Sensitivity (AI reflections correctly identified) and specificity (student reflections correctly identified) were then calculated, and the interview transcripts were analysed using thematic analysis.</p><p><strong>Results: </strong>Educators were unable to reliably distinguish between student and GenAI-authored reflections. Sensitivity across the four reflections ranged from 0.36 (95% CI: 0.16-0.61) to 0.64 (95% CI: 0.39-0.84). Specificity ranged from 0.64 (95% CI: 0.39-0.84) to 0.86 (95% CI: 0.60-0.96). Thematic analysis revealed three main themes when considering what features of the reflection educators used to make judgements about authorship: features of writing, features of reflection and educators' preconceptions and experiences.</p><p><strong>Discussion: </strong>This study demonstrates the challenges in differentiating between student- and GenAI-authored reflections, as well as highlighting the range of factors that influence this decision. Rather than developing ways to more accurately make this distinction or trying to stop students using GenAI, we suggest it could instead be harnessed to teach students reflective practice skills, and help students for whom written reflection in particular may be challenging.</p>\",\"PeriodicalId\":18370,\"journal\":{\"name\":\"Medical Education\",\"volume\":\" \",\"pages\":\"\"},\"PeriodicalIF\":5.2000,\"publicationDate\":\"2025-07-02\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Medical Education\",\"FirstCategoryId\":\"95\",\"ListUrlMain\":\"https://doi.org/10.1111/medu.15750\",\"RegionNum\":1,\"RegionCategory\":\"教育学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"EDUCATION, SCIENTIFIC DISCIPLINES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1111/medu.15750","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}
引用次数: 0

摘要

引言:反思是现代医生实践中不可或缺的一部分,虽然它可以采取多种形式,但在医学院的课程中,书面反思是很常见的。生成式人工智能(GenAI)越来越多地被使用,包括在医学课程中完成书面作业。我们试图探索教育工作者是否能够区分基因和学生撰写的反思,以及他们使用什么特征来做到这一点。方法:采用混合方法进行研究。28名教育工作者参加了一个“大声思考”的面试,并收到了四组反思,这些反思要么是由学生撰写的,要么是由GenAI撰写的,或者两者兼而有之。他们被要求辨认出他们认为是谁写的反思,同时大声说话。然后计算灵敏度(正确识别的人工智能反射)和特异性(正确识别的学生反射),并使用主题分析分析访谈记录。结果:教育工作者无法可靠地区分学生和genai撰写的反思。四种反射的灵敏度范围从0.36 (95% CI: 0.16-0.61)到0.64 (95% CI: 0.39-0.84)。特异性范围从0.64 (95% CI: 0.39-0.84)到0.86 (95% CI: 0.60-0.96)。主题分析揭示了三个主要主题,即:写作特征、反思特征和教育者的先入之见和经验。讨论:本研究展示了区分学生和genai撰写的反思所面临的挑战,并突出了影响这一决定的各种因素。与其开发更准确地区分这种区别的方法,或者试图阻止学生使用GenAI,我们建议可以利用它来教授学生反思练习技巧,并帮助那些对书面反思尤其具有挑战性的学生。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Can educators distinguish between medical student and generative AI-authored reflections?

Introduction: Reflection is integral to the modern doctor's practice and, whilst it can take many forms, written reflection is commonly found on medical school curricula. Generative artificial intelligence (GenAI) is increasingly being used, including in the completion of written assignments in medical curricula. We sought to explore if educators can distinguish between GenAI- and student-authored reflections and what features they use to do so.

Methods: This was a mixed-methods study. Twenty-eight educators attended a 'think aloud' interview and were presented with a set of four reflections, either all authored by students, all by GenAI or a mixture. They were asked to identify who they thought had written the reflection, speaking aloud whilst they did so. Sensitivity (AI reflections correctly identified) and specificity (student reflections correctly identified) were then calculated, and the interview transcripts were analysed using thematic analysis.

Results: Educators were unable to reliably distinguish between student and GenAI-authored reflections. Sensitivity across the four reflections ranged from 0.36 (95% CI: 0.16-0.61) to 0.64 (95% CI: 0.39-0.84). Specificity ranged from 0.64 (95% CI: 0.39-0.84) to 0.86 (95% CI: 0.60-0.96). Thematic analysis revealed three main themes when considering what features of the reflection educators used to make judgements about authorship: features of writing, features of reflection and educators' preconceptions and experiences.

Discussion: This study demonstrates the challenges in differentiating between student- and GenAI-authored reflections, as well as highlighting the range of factors that influence this decision. Rather than developing ways to more accurately make this distinction or trying to stop students using GenAI, we suggest it could instead be harnessed to teach students reflective practice skills, and help students for whom written reflection in particular may be challenging.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Medical Education
Medical Education 医学-卫生保健
CiteScore
8.40
自引率
10.00%
发文量
279
审稿时长
4-8 weeks
期刊介绍: Medical Education seeks to be the pre-eminent journal in the field of education for health care professionals, and publishes material of the highest quality, reflecting world wide or provocative issues and perspectives. The journal welcomes high quality papers on all aspects of health professional education including; -undergraduate education -postgraduate training -continuing professional development -interprofessional education
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信