评估人工智能在教育方面的能力：ChatGPT-4 在美国放射技师注册机构 (ARRT) 放射学认证考试中的表现。

IF 3.8 2区医学 Q1 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Academic Radiology Pub Date : 2025-02-01 DOI:10.1016/j.acra.2024.08.009

Yousif Al-Naser MRT(R) , Felobater Halka , Boris Ng BEng , Dwight Mountford MRT(R) MSc , Sonali Sharma , Ken Niure MRT(R) , Charlotte Yong-Hing MD , Faisal Khosa MD , Christian Van der Pol MD

{"title":"评估人工智能在教育方面的能力：ChatGPT-4 在美国放射技师注册机构 (ARRT) 放射学认证考试中的表现。","authors":"Yousif Al-Naser MRT(R) , Felobater Halka , Boris Ng BEng , Dwight Mountford MRT(R) MSc , Sonali Sharma , Ken Niure MRT(R) , Charlotte Yong-Hing MD , Faisal Khosa MD , Christian Van der Pol MD","doi":"10.1016/j.acra.2024.08.009","DOIUrl":null,"url":null,"abstract":"<div><h3>Rationale and Objectives</h3><div>The American Registry of Radiologic Technologists (ARRT) leads the certification process with an exam comprising 200 multiple-choice questions. This study aims to evaluate ChatGPT-4's performance in responding to practice questions similar to those found in the ARRT board examination.</div></div><div><h3>Materials and Methods</h3><div>We used a dataset of 200 practice multiple-choice questions for the ARRT certification exam from BoardVitals. Each question was fed to ChatGPT-4 fifteen times, resulting in 3000 observations to account for response variability.</div></div><div><h3>Results</h3><div>ChatGPT's overall performance was 80.56%, with higher accuracy on text-based questions (86.3%) compared to image-based questions (45.6%). Response times were longer for image-based questions (18.01 s) than for text-based questions (13.27 s). Performance varied by domain: 72.6% for Safety, 70.6% for Image Production, 67.3% for Patient Care, and 53.4% for Procedures. As anticipated, performance was best on on easy questions (78.5%).</div></div><div><h3>Conclusion</h3><div>ChatGPT demonstrated effective performance on the BoardVitals question bank for ARRT certification. Future studies could benefit from analyzing the correlation between BoardVitals scores and actual exam outcomes. Further development in AI, particularly in image processing and interpretation, is necessary to enhance its utility in educational settings.</div></div>","PeriodicalId":50928,"journal":{"name":"Academic Radiology","volume":"32 2","pages":"Pages 597-603"},"PeriodicalIF":3.8000,"publicationDate":"2025-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Evaluating Artificial Intelligence Competency in Education: Performance of ChatGPT-4 in the American Registry of Radiologic Technologists (ARRT) Radiography Certification Exam\",\"authors\":\"Yousif Al-Naser MRT(R) , Felobater Halka , Boris Ng BEng , Dwight Mountford MRT(R) MSc , Sonali Sharma , Ken Niure MRT(R) , Charlotte Yong-Hing MD , Faisal Khosa MD , Christian Van der Pol MD\",\"doi\":\"10.1016/j.acra.2024.08.009\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Rationale and Objectives</h3><div>The American Registry of Radiologic Technologists (ARRT) leads the certification process with an exam comprising 200 multiple-choice questions. This study aims to evaluate ChatGPT-4's performance in responding to practice questions similar to those found in the ARRT board examination.</div></div><div><h3>Materials and Methods</h3><div>We used a dataset of 200 practice multiple-choice questions for the ARRT certification exam from BoardVitals. Each question was fed to ChatGPT-4 fifteen times, resulting in 3000 observations to account for response variability.</div></div><div><h3>Results</h3><div>ChatGPT's overall performance was 80.56%, with higher accuracy on text-based questions (86.3%) compared to image-based questions (45.6%). Response times were longer for image-based questions (18.01 s) than for text-based questions (13.27 s). Performance varied by domain: 72.6% for Safety, 70.6% for Image Production, 67.3% for Patient Care, and 53.4% for Procedures. As anticipated, performance was best on on easy questions (78.5%).</div></div><div><h3>Conclusion</h3><div>ChatGPT demonstrated effective performance on the BoardVitals question bank for ARRT certification. Future studies could benefit from analyzing the correlation between BoardVitals scores and actual exam outcomes. Further development in AI, particularly in image processing and interpretation, is necessary to enhance its utility in educational settings.</div></div>\",\"PeriodicalId\":50928,\"journal\":{\"name\":\"Academic Radiology\",\"volume\":\"32 2\",\"pages\":\"Pages 597-603\"},\"PeriodicalIF\":3.8000,\"publicationDate\":\"2025-02-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Academic Radiology\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1076633224005725\",\"RegionNum\":2,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Academic Radiology","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1076633224005725","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

摘要

理由和目标：美国放射技师注册委员会 (ARRT) 主导认证程序，考试包括 200 道选择题。本研究旨在评估 ChatGPT-4 在回答与 ARRT 委员会考试类似的练习题时的表现：我们使用了 BoardVitals 提供的 200 道 ARRT 认证考试练习选择题数据集。每道题都向 ChatGPT-4 发送了 15 次，共观察了 3000 次，以考虑反应的可变性：ChatGPT 的总体性能为 80.56%，其中文本问题（86.3%）的准确率高于图像问题（45.6%）。图像问题的回复时间（18.01 秒）比文本问题的回复时间（13.27 秒）长。不同领域的成绩各不相同：安全 72.6%，图像制作 70.6%，病人护理 67.3%，程序 53.4%。正如预期的那样，简单问题的成绩最好（78.5%）：ChatGPT 在 ARRT 认证的 BoardVitals 题库中表现出色。分析 BoardVitals 分数与实际考试结果之间的相关性将有助于今后的研究。有必要进一步发展人工智能，尤其是在图像处理和解读方面，以提高其在教育环境中的实用性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Evaluating Artificial Intelligence Competency in Education: Performance of ChatGPT-4 in the American Registry of Radiologic Technologists (ARRT) Radiography Certification Exam

Rationale and Objectives

The American Registry of Radiologic Technologists (ARRT) leads the certification process with an exam comprising 200 multiple-choice questions. This study aims to evaluate ChatGPT-4's performance in responding to practice questions similar to those found in the ARRT board examination.

Materials and Methods

We used a dataset of 200 practice multiple-choice questions for the ARRT certification exam from BoardVitals. Each question was fed to ChatGPT-4 fifteen times, resulting in 3000 observations to account for response variability.

Results

ChatGPT's overall performance was 80.56%, with higher accuracy on text-based questions (86.3%) compared to image-based questions (45.6%). Response times were longer for image-based questions (18.01 s) than for text-based questions (13.27 s). Performance varied by domain: 72.6% for Safety, 70.6% for Image Production, 67.3% for Patient Care, and 53.4% for Procedures. As anticipated, performance was best on on easy questions (78.5%).

Conclusion

ChatGPT demonstrated effective performance on the BoardVitals question bank for ARRT certification. Future studies could benefit from analyzing the correlation between BoardVitals scores and actual exam outcomes. Further development in AI, particularly in image processing and interpretation, is necessary to enhance its utility in educational settings.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Academic Radiology 医学-核医学

CiteScore

7.60

自引率

10.40%

发文量

432

审稿时长

18 days

期刊介绍： Academic Radiology publishes original reports of clinical and laboratory investigations in diagnostic imaging, the diagnostic use of radioactive isotopes, computed tomography, positron emission tomography, magnetic resonance imaging, ultrasound, digital subtraction angiography, image-guided interventions and related techniques. It also includes brief technical reports describing original observations, techniques, and instrumental developments; state-of-the-art reports on clinical issues, new technology and other topics of current medical importance; meta-analyses; scientific studies and opinions on radiologic education; and letters to the Editor.