Diane Ghanem , Joseph E. Nassar , Joseph El Bachour , Tammam Hanna
{"title":"ChatGPT Earns American Board Certification in Hand Surgery","authors":"Diane Ghanem , Joseph E. Nassar , Joseph El Bachour , Tammam Hanna","doi":"10.1016/j.hansur.2024.101688","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><p>Artificial Intelligence (AI), and specifically ChatGPT, has shown potential in healthcare, yet its performance in specialized medical examinations such as the Orthopaedic Surgery In-Training Examination and European Board Hand Surgery diploma has been inconsistent. This study aims to evaluate the capability of ChatGPT-4 to pass the American Hand Surgery Certifying Examination.</p></div><div><h3>Methods</h3><p>ChatGPT-4 was tested on the 2019 American Society for Surgery of the Hand (ASSH) Self-Assessment Exam. All 200 questions available online (<span>https://onlinecme.assh.org</span><svg><path></path></svg>) were retrieved. All media-containing questions were flagged and carefully reviewed. Eight media-containing questions were excluded as they either relied purely on videos or could not be rationalized from the presented information. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT-4. The ASSH report was used to compare ChatGPT-4’s performance to that of the 322 physicians who completed the 2019 ASSH self-assessment.</p></div><div><h3>Results</h3><p>ChatGPT-4 answered 192 questions with an overall score of 61.98%. Performance on media-containing questions was 55.56%, while on non-media questions it was 65.83%, with no statistical difference in performance based on media inclusion. Despite scoring below the average physician’s performance, ChatGPT-4 outperformed in the ‘vascular’ section with 81.82%. Its performance was lower in the ‘bone and joint’ (48.54%) and ‘neuromuscular’ (56.25%) sections.</p></div><div><h3>Conclusions</h3><p>ChatGPT-4 achieved a good overall score of 61.98%. This AI language model demonstrates significant capability in processing and answering specialized medical examination questions, albeit with room for improvement in areas requiring complex clinical judgment and nuanced interpretation. ChatGPT-4’s proficiency is influenced by the structure and language of the examination, with no replacement for the depth of trained medical specialists. This study underscores the supportive role of AI in medical education and clinical decision-making while highlighting the current limitations in nuanced fields such as hand surgery.</p></div>","PeriodicalId":54301,"journal":{"name":"Hand Surgery & Rehabilitation","volume":"43 3","pages":"Article 101688"},"PeriodicalIF":0.9000,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Hand Surgery & Rehabilitation","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2468122924000653","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"ORTHOPEDICS","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose
Artificial Intelligence (AI), and specifically ChatGPT, has shown potential in healthcare, yet its performance in specialized medical examinations such as the Orthopaedic Surgery In-Training Examination and European Board Hand Surgery diploma has been inconsistent. This study aims to evaluate the capability of ChatGPT-4 to pass the American Hand Surgery Certifying Examination.
Methods
ChatGPT-4 was tested on the 2019 American Society for Surgery of the Hand (ASSH) Self-Assessment Exam. All 200 questions available online (https://onlinecme.assh.org) were retrieved. All media-containing questions were flagged and carefully reviewed. Eight media-containing questions were excluded as they either relied purely on videos or could not be rationalized from the presented information. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT-4. The ASSH report was used to compare ChatGPT-4’s performance to that of the 322 physicians who completed the 2019 ASSH self-assessment.
Results
ChatGPT-4 answered 192 questions with an overall score of 61.98%. Performance on media-containing questions was 55.56%, while on non-media questions it was 65.83%, with no statistical difference in performance based on media inclusion. Despite scoring below the average physician’s performance, ChatGPT-4 outperformed in the ‘vascular’ section with 81.82%. Its performance was lower in the ‘bone and joint’ (48.54%) and ‘neuromuscular’ (56.25%) sections.
Conclusions
ChatGPT-4 achieved a good overall score of 61.98%. This AI language model demonstrates significant capability in processing and answering specialized medical examination questions, albeit with room for improvement in areas requiring complex clinical judgment and nuanced interpretation. ChatGPT-4’s proficiency is influenced by the structure and language of the examination, with no replacement for the depth of trained medical specialists. This study underscores the supportive role of AI in medical education and clinical decision-making while highlighting the current limitations in nuanced fields such as hand surgery.
期刊介绍：
As the official publication of the French, Belgian and Swiss Societies for Surgery of the Hand, as well as of the French Society of Rehabilitation of the Hand & Upper Limb, ''Hand Surgery and Rehabilitation'' - formerly named "Chirurgie de la Main" - publishes original articles, literature reviews, technical notes, and clinical cases. It is indexed in the main international databases (including Medline). Initially a platform for French-speaking hand surgeons, the journal will now publish its articles in English to disseminate its author''s scientific findings more widely. The journal also includes a biannual supplement in French, the monograph of the French Society for Surgery of the Hand, where comprehensive reviews in the fields of hand, peripheral nerve and upper limb surgery are presented.
Organe officiel de la Société française de chirurgie de la main, de la Société française de Rééducation de la main (SFRM-GEMMSOR), de la Société suisse de chirurgie de la main et du Belgian Hand Group, indexée dans les grandes bases de données internationales (Medline, Embase, Pascal, Scopus), Hand Surgery and Rehabilitation - anciennement titrée Chirurgie de la main - publie des articles originaux, des revues de la littérature, des notes techniques, des cas clinique. Initialement plateforme d''expression francophone de la spécialité, la revue s''oriente désormais vers l''anglais pour devenir une référence scientifique et de formation de la spécialité en France et en Europe. Avec 6 publications en anglais par an, la revue comprend également un supplément biannuel, la monographie du GEM, où sont présentées en français, des mises au point complètes dans les domaines de la chirurgie de la main, des nerfs périphériques et du membre supérieur.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
