在放射科住院医师申请中使用大型语言模型：不受欢迎但不可避免。

IF 4 3区医学 Q1 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Journal of the American College of Radiology Pub Date : 2025-01-01 DOI:10.1016/j.jacr.2024.08.027

Emile B. Gordon MD , Charles M. Maxfield MD , Robert French MD, MBA , Laura J. Fish MD , Jacob Romm MD , Emily Barre MPH, BSE , Erica Kinne MD , Ryan Peterson MD , Lars J. Grimm MD, MHS

{"title":"在放射科住院医师申请中使用大型语言模型：不受欢迎但不可避免。","authors":"Emile B. Gordon MD , Charles M. Maxfield MD , Robert French MD, MBA , Laura J. Fish MD , Jacob Romm MD , Emily Barre MPH, BSE , Erica Kinne MD , Ryan Peterson MD , Lars J. Grimm MD, MHS","doi":"10.1016/j.jacr.2024.08.027","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>This study explores radiology program directors’ perspectives on the impact of large language model (LLM) use among residency applicants to craft personal statements.</div></div><div><h3>Methods</h3><div>Eight program directors from the Radiology Residency Education Research Alliance participated in a mixed-methods study, which included a survey regarding impressions of artificial intelligence (AI)-generated personal statements and focus group discussions (July 2023). Each director reviewed four personal statement variations for five applicants, anonymized to author type: the original and three Chat Generative Pre-trained Transformer-4.0 (GPT) versions generated with varying prompts, aggregated for analysis. A 5-point Likert scale surveyed the writing quality, including voice, clarity, engagement, organization, and perceived origin of each statement. An experienced qualitative researcher facilitated focus group discussions. Data analysis was performed using a rapid analytic approach with a coding template capturing key areas related to residency applications.</div></div><div><h3>Results</h3><div>GPT-generated statement ratings were more often average or worse in quality (56%, 268 of 475) than ratings of human-authored statements (29%, 45 of 160). Although reviewers were not confident in their ability to distinguish the origin of personal statements, they did so reliably and consistently, identifying the human-authored personal statements at 95% (38 of 40) as probably or definitely original. Focus group discussions highlighted the inevitable use of AI in crafting personal statements and concerns about its impact on the authenticity and the value of the personal statement in residency selections. Program directors were divided on the appropriate use and regulation of AI.</div></div><div><h3>Discussion</h3><div>Radiology residency program directors rated LLM-generated personal statements as lower in quality and expressed concern about the loss of the applicant’s voice but acknowledged the inevitability of increased AI use in the generation of application statements.</div></div>","PeriodicalId":49044,"journal":{"name":"Journal of the American College of Radiology","volume":"22 1","pages":"Pages 33-40"},"PeriodicalIF":4.0000,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Large Language Model Use in Radiology Residency Applications: Unwelcomed but Inevitable\",\"authors\":\"Emile B. Gordon MD , Charles M. Maxfield MD , Robert French MD, MBA , Laura J. Fish MD , Jacob Romm MD , Emily Barre MPH, BSE , Erica Kinne MD , Ryan Peterson MD , Lars J. Grimm MD, MHS\",\"doi\":\"10.1016/j.jacr.2024.08.027\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Objective</h3><div>This study explores radiology program directors’ perspectives on the impact of large language model (LLM) use among residency applicants to craft personal statements.</div></div><div><h3>Methods</h3><div>Eight program directors from the Radiology Residency Education Research Alliance participated in a mixed-methods study, which included a survey regarding impressions of artificial intelligence (AI)-generated personal statements and focus group discussions (July 2023). Each director reviewed four personal statement variations for five applicants, anonymized to author type: the original and three Chat Generative Pre-trained Transformer-4.0 (GPT) versions generated with varying prompts, aggregated for analysis. A 5-point Likert scale surveyed the writing quality, including voice, clarity, engagement, organization, and perceived origin of each statement. An experienced qualitative researcher facilitated focus group discussions. Data analysis was performed using a rapid analytic approach with a coding template capturing key areas related to residency applications.</div></div><div><h3>Results</h3><div>GPT-generated statement ratings were more often average or worse in quality (56%, 268 of 475) than ratings of human-authored statements (29%, 45 of 160). Although reviewers were not confident in their ability to distinguish the origin of personal statements, they did so reliably and consistently, identifying the human-authored personal statements at 95% (38 of 40) as probably or definitely original. Focus group discussions highlighted the inevitable use of AI in crafting personal statements and concerns about its impact on the authenticity and the value of the personal statement in residency selections. Program directors were divided on the appropriate use and regulation of AI.</div></div><div><h3>Discussion</h3><div>Radiology residency program directors rated LLM-generated personal statements as lower in quality and expressed concern about the loss of the applicant’s voice but acknowledged the inevitability of increased AI use in the generation of application statements.</div></div>\",\"PeriodicalId\":49044,\"journal\":{\"name\":\"Journal of the American College of Radiology\",\"volume\":\"22 1\",\"pages\":\"Pages 33-40\"},\"PeriodicalIF\":4.0000,\"publicationDate\":\"2025-01-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of the American College of Radiology\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1546144024007683\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the American College of Radiology","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1546144024007683","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

摘要

目的本研究探讨放射学项目主任对住院医师申请者使用大语言模型（LLM）撰写个人陈述的影响的看法：来自放射学住院医师教育研究联盟（RRERA）的八位项目主任参与了一项混合方法研究，其中包括一项关于人工智能生成的个人陈述印象的调查和焦点小组讨论（2023 年 7 月）。每位主任审查了五位申请人的四份个人陈述变体，并对作者类型进行了盲审：原始版本和根据不同提示生成的三个 ChatGPT-4.0 版本，汇总后进行分析。采用 5 分李克特量表对写作质量进行调查，包括语音、清晰度、参与度、条理性以及每份陈述的感知来源。一位经验丰富的定性研究人员主持了焦点小组讨论。数据分析采用快速分析方法进行，编码模板捕捉了与住院实习申请相关的关键领域：结果：GPT生成的声明（GPT）的评分质量一般或更差（56%，268/475），高于人类撰写的声明（Hu）的评分（29% [45/160]）。尽管审稿人对自己辨别个人陈述来源的能力并不自信，但他们的辨别能力是可靠和一致的，95%（38/40）的人撰个人陈述可能或肯定是原创的。焦点小组讨论强调了在撰写个人陈述时不可避免地使用人工智能的问题，以及人工智能对个人陈述的真实性和在住院医生遴选中的价值所产生的影响。项目主任们对人工智能的适当使用和监管存在分歧：放射科住院医师培训项目主任认为由 LLM 生成的个人陈述质量较低，并对失去申请人的声音表示担忧，但也承认在生成申请陈述时越来越多地使用人工智能是不可避免的。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Large Language Model Use in Radiology Residency Applications: Unwelcomed but Inevitable

Objective

This study explores radiology program directors’ perspectives on the impact of large language model (LLM) use among residency applicants to craft personal statements.

Methods

Eight program directors from the Radiology Residency Education Research Alliance participated in a mixed-methods study, which included a survey regarding impressions of artificial intelligence (AI)-generated personal statements and focus group discussions (July 2023). Each director reviewed four personal statement variations for five applicants, anonymized to author type: the original and three Chat Generative Pre-trained Transformer-4.0 (GPT) versions generated with varying prompts, aggregated for analysis. A 5-point Likert scale surveyed the writing quality, including voice, clarity, engagement, organization, and perceived origin of each statement. An experienced qualitative researcher facilitated focus group discussions. Data analysis was performed using a rapid analytic approach with a coding template capturing key areas related to residency applications.

Results

GPT-generated statement ratings were more often average or worse in quality (56%, 268 of 475) than ratings of human-authored statements (29%, 45 of 160). Although reviewers were not confident in their ability to distinguish the origin of personal statements, they did so reliably and consistently, identifying the human-authored personal statements at 95% (38 of 40) as probably or definitely original. Focus group discussions highlighted the inevitable use of AI in crafting personal statements and concerns about its impact on the authenticity and the value of the personal statement in residency selections. Program directors were divided on the appropriate use and regulation of AI.

Discussion

Radiology residency program directors rated LLM-generated personal statements as lower in quality and expressed concern about the loss of the applicant’s voice but acknowledged the inevitability of increased AI use in the generation of application statements.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Journal of the American College of Radiology RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING-

CiteScore

6.30

自引率

8.90%

发文量

312

审稿时长

34 days

期刊介绍： The official journal of the American College of Radiology, JACR informs its readers of timely, pertinent, and important topics affecting the practice of diagnostic radiologists, interventional radiologists, medical physicists, and radiation oncologists. In so doing, JACR improves their practices and helps optimize their role in the health care system. By providing a forum for informative, well-written articles on health policy, clinical practice, practice management, data science, and education, JACR engages readers in a dialogue that ultimately benefits patient care.