Artificial Intelligence Efficacy as a Function of Trainee Interpreter Proficiency: Lessons from a Randomized Controlled Trial.

David A Fussell, Cynthia C Tang, Jake Sternhagen, Varun V Marrey, Kelsey M Roman, Jeremy Johnson, Michael J Head, Hayden R Troutt, Charles H Li, Peter D Chang, John Joseph, Daniel S Chow
{"title":"Artificial Intelligence Efficacy as a Function of Trainee Interpreter Proficiency: Lessons from a Randomized Controlled Trial.","authors":"David A Fussell, Cynthia C Tang, Jake Sternhagen, Varun V Marrey, Kelsey M Roman, Jeremy Johnson, Michael J Head, Hayden R Troutt, Charles H Li, Peter D Chang, John Joseph, Daniel S Chow","doi":"10.3174/ajnr.A8387","DOIUrl":null,"url":null,"abstract":"<p><strong>Background and purpose: </strong>Recently, artificial intelligence tools have been deployed with increasing speed in educational and clinical settings. However, the use of artificial intelligence by trainees across different levels of experience has not been well-studied. This study investigates the impact of artificial intelligence assistance on the diagnostic accuracy for intracranial hemorrhage and large-vessel occlusion by medical students and resident trainees.</p><p><strong>Materials and methods: </strong>This prospective study was conducted between March 2023 and October 2023. Medical students and resident trainees were asked to identify intracranial hemorrhage and large-vessel occlusion in 100 noncontrast head CTs and 100 head CTAs, respectively. One group received diagnostic aid simulating artificial intelligence for intracranial hemorrhage only (<i>n</i> = 26); the other, for large-vessel occlusion only (<i>n</i> = 28). Primary outcomes included accuracy, sensitivity, and specificity for intracranial hemorrhage/large-vessel occlusion detection without and with aid. Study interpretation time was a secondary outcome. Individual responses were pooled and analyzed with the <i>t</i> test; differences in continuous variables were assessed with ANOVA.</p><p><strong>Results: </strong>Forty-eight participants completed the study, generating 10,779 intracranial hemorrhage or large-vessel occlusion interpretations. With diagnostic aid, medical student accuracy improved 11.0 points (<i>P</i> < .001) and resident trainee accuracy showed no significant change. Intracranial hemorrhage interpretation time increased with diagnostic aid for both groups (<i>P</i> < .001), while large-vessel occlusion interpretation time decreased for medical students (<i>P</i> < .001). Despite worse performance in the detection of the smallest-versus-largest hemorrhages at baseline, medical students were not more likely to accept a true-positive artificial intelligence result for these more difficult tasks. Both groups were considerably less accurate when disagreeing with the artificial intelligence or when supplied with an incorrect artificial intelligence result.</p><p><strong>Conclusions: </strong>This study demonstrated greater improvement in diagnostic accuracy with artificial intelligence for medical students compared with resident trainees. However, medical students were less likely than resident trainees to overrule incorrect artificial intelligence interpretations and were less accurate, even with diagnostic aid, than the artificial intelligence was by itself.</p>","PeriodicalId":93863,"journal":{"name":"AJNR. American journal of neuroradiology","volume":" ","pages":"1647-1654"},"PeriodicalIF":0.0000,"publicationDate":"2024-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11543080/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"AJNR. American journal of neuroradiology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3174/ajnr.A8387","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Background and purpose: Recently, artificial intelligence tools have been deployed with increasing speed in educational and clinical settings. However, the use of artificial intelligence by trainees across different levels of experience has not been well-studied. This study investigates the impact of artificial intelligence assistance on the diagnostic accuracy for intracranial hemorrhage and large-vessel occlusion by medical students and resident trainees.

Materials and methods: This prospective study was conducted between March 2023 and October 2023. Medical students and resident trainees were asked to identify intracranial hemorrhage and large-vessel occlusion in 100 noncontrast head CTs and 100 head CTAs, respectively. One group received diagnostic aid simulating artificial intelligence for intracranial hemorrhage only (n = 26); the other, for large-vessel occlusion only (n = 28). Primary outcomes included accuracy, sensitivity, and specificity for intracranial hemorrhage/large-vessel occlusion detection without and with aid. Study interpretation time was a secondary outcome. Individual responses were pooled and analyzed with the t test; differences in continuous variables were assessed with ANOVA.

Results: Forty-eight participants completed the study, generating 10,779 intracranial hemorrhage or large-vessel occlusion interpretations. With diagnostic aid, medical student accuracy improved 11.0 points (P < .001) and resident trainee accuracy showed no significant change. Intracranial hemorrhage interpretation time increased with diagnostic aid for both groups (P < .001), while large-vessel occlusion interpretation time decreased for medical students (P < .001). Despite worse performance in the detection of the smallest-versus-largest hemorrhages at baseline, medical students were not more likely to accept a true-positive artificial intelligence result for these more difficult tasks. Both groups were considerably less accurate when disagreeing with the artificial intelligence or when supplied with an incorrect artificial intelligence result.

Conclusions: This study demonstrated greater improvement in diagnostic accuracy with artificial intelligence for medical students compared with resident trainees. However, medical students were less likely than resident trainees to overrule incorrect artificial intelligence interpretations and were less accurate, even with diagnostic aid, than the artificial intelligence was by itself.

人工智能功效与受训口译员能力的关系:随机对照试验的启示。
背景和目的:最近,人工智能工具在教育和临床环境中的应用速度越来越快。然而,对不同经验水平的学员使用人工智能的情况还没有进行深入研究。本研究调查了人工智能辅助工具对医科学生(MS)和住院医师培训生(RT)颅内出血(ICH)和大血管闭塞(LVO)诊断准确性的影响:这项前瞻性研究在 2023 年 3 月至 2023 年 10 月期间进行。要求 MS 和 RT 分别在 100 张非对比头部 CT 和 100 张头部 CTA 中识别 ICH 和 LVO。其中一组只接受模拟 AI 的 ICH 诊断辅助(26 人),另一组只接受模拟 AI 的 LVO 诊断辅助(28 人)。主要结果包括无辅助和有辅助时检测 ICH / LVO 的准确性、灵敏度和特异性。研究解释时间是次要结果。对个人反应进行汇总,并用卡方进行分析;连续变量的差异用方差分析进行评估:48名参与者完成了研究,共进行了10779次ICH或LVO解读。使用诊断辅助工具后,MS 的准确性提高了 11.0 分(P < .001),RT 的准确性没有明显变化。使用诊断辅助工具后,两组的 ICH 解读时间均有所增加(P < .001),而 MS 的 LVO 解读时间则有所减少(P < .001)。尽管在基线时,MS 在检测最小出血和最大出血方面的表现较差,但在这些难度较大的任务中,MS 接受 AI 真阳性结果的可能性并不大。在不同意人工智能结果或提供错误人工智能结果时,两组人的准确性都要低得多:本研究表明,与 RT 相比,MS 使用人工智能诊断的准确性有了更大的提高。然而,与 RT 相比,MS 更不可能推翻不正确的 AI 解释,即使使用诊断辅助工具,其准确性也不如 AI 本身:缩写:ICH=颅内出血;LVO=大血管闭塞;MS=医科学生;RT=住院受训人员。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信