ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags.

IF 1.4 Q3 PEDIATRICS

Global Pediatric Health Pub Date : 2024-03-24 eCollection Date: 2024-01-01 DOI:10.1177/2333794X241240327

Mindy Le, Michael Davis

{"title":"ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags.","authors":"Mindy Le, Michael Davis","doi":"10.1177/2333794X241240327","DOIUrl":null,"url":null,"abstract":"Objectives: We aimed to evaluate the performance of a publicly-available online artificial intelligence program (OpenAI's ChatGPT-3.5 and -4.0, August 3 versions) on a pediatric board preparatory examination, 2021 and 2022 PREP® Self-Assessment, American Academy of Pediatrics (AAP).Methods: We entered 245 questions and answer choices from the Pediatrics 2021 PREP® Self-Assessment and 247 questions and answer choices from the Pediatrics 2022 PREP® Self-Assessment into OpenAI's ChatGPT-3.5 and ChatGPT-4.0, August 3 versions, in September 2023. The ChatGPT-3.5 and 4.0 scores were compared with the advertised passing scores (70%+) for the PREP® exams and the average scores (74.09%) and (75.71%) for all 10 715 and 6825 first-time human test takers.Results: For the AAP 2021 and 2022 PREP® Self-Assessments, ChatGPT-3.5 answered 143 of 243 (58.85%) and 137 of 247 (55.46%) questions correctly on a single attempt. ChatGPT-4.0 answered 193 of 243 (79.84%) and 208 of 247 (84.21%) questions correctly.Conclusion: Using a publicly-available online chatbot to answer pediatric board preparatory examination questions yielded a passing score but demonstrated significant limitations in the chatbot's ability to assess some complex medical situations in children, posing a potential risk to this vulnerable population.","PeriodicalId":12576,"journal":{"name":"Global Pediatric Health","volume":null,"pages":null},"PeriodicalIF":1.4000,"publicationDate":"2024-03-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10962030/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Global Pediatric Health","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1177/2333794X241240327","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"Q3","JCRName":"PEDIATRICS","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives: We aimed to evaluate the performance of a publicly-available online artificial intelligence program (OpenAI's ChatGPT-3.5 and -4.0, August 3 versions) on a pediatric board preparatory examination, 2021 and 2022 PREP^® Self-Assessment, American Academy of Pediatrics (AAP).

Methods: We entered 245 questions and answer choices from the Pediatrics 2021 PREP^® Self-Assessment and 247 questions and answer choices from the Pediatrics 2022 PREP^® Self-Assessment into OpenAI's ChatGPT-3.5 and ChatGPT-4.0, August 3 versions, in September 2023. The ChatGPT-3.5 and 4.0 scores were compared with the advertised passing scores (70%+) for the PREP^® exams and the average scores (74.09%) and (75.71%) for all 10 715 and 6825 first-time human test takers.

Results: For the AAP 2021 and 2022 PREP^® Self-Assessments, ChatGPT-3.5 answered 143 of 243 (58.85%) and 137 of 247 (55.46%) questions correctly on a single attempt. ChatGPT-4.0 answered 193 of 243 (79.84%) and 208 of 247 (84.21%) questions correctly.

Conclusion: Using a publicly-available online chatbot to answer pediatric board preparatory examination questions yielded a passing score but demonstrated significant limitations in the chatbot's ability to assess some complex medical situations in children, posing a potential risk to this vulnerable population.

查看原文本刊更多论文

ChatGPT 在儿科医师资格预备考试中获得及格分数，但却引起了警惕。

研究目的我们旨在评估公开在线人工智能程序（OpenAI的ChatGPT-3.5和-4.0，8月3日版本）在美国儿科学会（AAP）2021年和2022年PREP®自我评估儿科委员会预备考试中的表现：我们于 2023 年 9 月将儿科 2021 年 PREP® 自我评估中的 245 个问题和答案选择以及儿科 2022 年 PREP® 自我评估中的 247 个问题和答案选择输入 OpenAI 的 ChatGPT-3.5 和 ChatGPT-4.0（8 月 3 日版本）。将 ChatGPT-3.5 和 4.0 分数与 PREP® 考试的广告合格分数（70% 以上）以及所有 10 715 名和 6825 名首次参加人类考试者的平均分数（74.09%）和（75.71%）进行了比较：对于 AAP 2021 和 2022 PREP® 自我评估，ChatGPT-3.5 一次就正确回答了 243 道题中的 143 道题（58.85%）和 247 道题中的 137 道题（55.46%）。ChatGPT-4.0 能正确回答 243 个问题中的 193 个（79.84%）和 247 个问题中的 208 个（84.21%）：使用公开的在线聊天机器人回答儿科医师资格考试的预备考试问题可获得及格分数，但聊天机器人在评估儿童某些复杂医疗情况的能力方面存在很大的局限性，这给这一弱势群体带来了潜在的风险。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Global Pediatric Health Nursing-Pediatrics

CiteScore

2.20

自引率

0.00%

发文量

105

审稿时长

12 weeks