Exploring the potential of ChatGPT for clinical reasoning and decision-making: a cross-sectional study on the Italian Medical Residency Exam.

IF 1.1 4区医学 Q4 PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH

Annali dell'Istituto superiore di sanita Pub Date : 2023-10-01 DOI:10.4415/ANN_23_04_05

Giacomo Scaioli, Giuseppina Lo Moro, Francesco Conrado, Lorenzo Rosset, Fabrizio Bert, Roberta Siliquini

{"title":"Exploring the potential of ChatGPT for clinical reasoning and decision-making: a cross-sectional study on the Italian Medical Residency Exam.","authors":"Giacomo Scaioli, Giuseppina Lo Moro, Francesco Conrado, Lorenzo Rosset, Fabrizio Bert, Roberta Siliquini","doi":"10.4415/ANN_23_04_05","DOIUrl":null,"url":null,"abstract":"Background: This study aimed to assess the performance of ChatGPT, a large language model (LLM), on the Italian State Exam for Medical Residency (SSM) test to determine its potential as a tool for medical education and clinical decision-making support.Materials and methods: A total of 136 questions were obtained from the official SSM test. ChatGPT responses were analyzed and compared to the performance of medical doctors who took the test in 2022. Questions were classified into clinical cases (CC) and notional questions (NQ).Results: ChatGPT achieved an overall accuracy of 90.44%, with higher performance on clinical cases (92.45%) than on notional questions (89.15%). Compared to medical doctors' scores, ChatGPT performance was higher than 99.6% of the participants.Conclusions: These results suggest that ChatGPT holds promise as a valuable tool in clinical decision-making, particularly in the context of clinical reasoning. Further research is needed to explore the potential applications and implementation of large language models (LLMs) in medical education and medical practice.","PeriodicalId":8246,"journal":{"name":"Annali dell'Istituto superiore di sanita","volume":null,"pages":null},"PeriodicalIF":1.1000,"publicationDate":"2023-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Annali dell'Istituto superiore di sanita","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.4415/ANN_23_04_05","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}

引用次数: 0

Abstract

Background: This study aimed to assess the performance of ChatGPT, a large language model (LLM), on the Italian State Exam for Medical Residency (SSM) test to determine its potential as a tool for medical education and clinical decision-making support.

Materials and methods: A total of 136 questions were obtained from the official SSM test. ChatGPT responses were analyzed and compared to the performance of medical doctors who took the test in 2022. Questions were classified into clinical cases (CC) and notional questions (NQ).

Results: ChatGPT achieved an overall accuracy of 90.44%, with higher performance on clinical cases (92.45%) than on notional questions (89.15%). Compared to medical doctors' scores, ChatGPT performance was higher than 99.6% of the participants.

Conclusions: These results suggest that ChatGPT holds promise as a valuable tool in clinical decision-making, particularly in the context of clinical reasoning. Further research is needed to explore the potential applications and implementation of large language models (LLMs) in medical education and medical practice.

查看原文本刊更多论文

探索 ChatGPT 在临床推理和决策方面的潜力：一项关于意大利医学住院医师考试的横断面研究。

研究背景本研究旨在评估大语言模型（LLM）ChatGPT 在意大利国家住院医师考试（SSM）测试中的表现，以确定其作为医学教育和临床决策支持工具的潜力：从官方的 SSM 考试中总共获得了 136 个问题。对 ChatGPT 的回答进行了分析，并与 2022 年参加考试的医生的表现进行了比较。问题分为临床病例（CC）和概念问题（NQ）：结果：ChatGPT 的总体准确率为 90.44%，其中临床病例的准确率（92.45%）高于概念性问题的准确率（89.15%）。与医生的评分相比，ChatGPT 的表现高于 99.6% 的参与者：这些结果表明，ChatGPT 有望成为临床决策的重要工具，尤其是在临床推理方面。还需要进一步研究探索大语言模型（LLM）在医学教育和医疗实践中的潜在应用和实施。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Annali dell'Istituto superiore di sanita PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH-

CiteScore

3.80

自引率

4.80%

发文量

审稿时长

>12 weeks

期刊介绍： Annali dell’Istituto Superiore di Sanità is a peer reviewed quarterly science journal which publishes research articles in biomedicine, translational research and in many other disciplines of the health sciences. The journal includes the following material: original articles, reviews, commentaries, editorials, brief and technical notes, book reviews. The publication of Monographic Sections has been discontinued. In case you wish to present a small number of coordinated contributions on specific themes concerning priorities in public health, please contact the Editorial office. The journal is in English.