Assessing ChatGPT's theoretical knowledge and prescriptive accuracy in bacterial infections: a comparative study with infectious diseases residents and specialists.
Andrea De Vito, Nicholas Geremia, Andrea Marino, Davide Fiore Bavaro, Giorgia Caruana, Marianna Meschiari, Agnese Colpani, Maria Mazzitelli, Vincenzo Scaglione, Emmanuele Venanzi Rullo, Vito Fiore, Marco Fois, Edoardo Campanella, Eugenia Pistarà, Matteo Faltoni, Giuseppe Nunnari, Annamaria Cattelan, Cristina Mussini, Michele Bartoletti, Luigi Angelo Vaira, Giordano Madeddu
{"title":"Assessing ChatGPT's theoretical knowledge and prescriptive accuracy in bacterial infections: a comparative study with infectious diseases residents and specialists.","authors":"Andrea De Vito, Nicholas Geremia, Andrea Marino, Davide Fiore Bavaro, Giorgia Caruana, Marianna Meschiari, Agnese Colpani, Maria Mazzitelli, Vincenzo Scaglione, Emmanuele Venanzi Rullo, Vito Fiore, Marco Fois, Edoardo Campanella, Eugenia Pistarà, Matteo Faltoni, Giuseppe Nunnari, Annamaria Cattelan, Cristina Mussini, Michele Bartoletti, Luigi Angelo Vaira, Giordano Madeddu","doi":"10.1007/s15010-024-02350-6","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Advancements in Artificial Intelligence(AI) have made platforms like ChatGPT increasingly relevant in medicine. This study assesses ChatGPT's utility in addressing bacterial infection-related questions and antibiogram-based clinical cases.</p><p><strong>Methods: </strong>This study involved a collaborative effort involving infectious disease (ID) specialists and residents. A group of experts formulated six true/false, six open-ended questions, and six clinical cases with antibiograms for four types of infections (endocarditis, pneumonia, intra-abdominal infections, and bloodstream infection) for a total of 96 questions. The questions were submitted to four senior residents and four specialists in ID and inputted into ChatGPT-4 and a trained version of ChatGPT-4. A total of 720 responses were obtained and reviewed by a blinded panel of experts in antibiotic treatments. They evaluated the responses for accuracy and completeness, the ability to identify correct resistance mechanisms from antibiograms, and the appropriateness of antibiotics prescriptions.</p><p><strong>Results: </strong>No significant difference was noted among the four groups for true/false questions, with approximately 70% correct answers. The trained ChatGPT-4 and ChatGPT-4 offered more accurate and complete answers to the open-ended questions than both the residents and specialists. Regarding the clinical case, we observed a lower accuracy from ChatGPT-4 to recognize the correct resistance mechanism. ChatGPT-4 tended not to prescribe newer antibiotics like cefiderocol or imipenem/cilastatin/relebactam, favoring less recommended options like colistin. Both trained- ChatGPT-4 and ChatGPT-4 recommended longer than necessary treatment periods (p-value = 0.022).</p><p><strong>Conclusions: </strong>This study highlights ChatGPT's capabilities and limitations in medical decision-making, specifically regarding bacterial infections and antibiogram analysis. While ChatGPT demonstrated proficiency in answering theoretical questions, it did not consistently align with expert decisions in clinical case management. Despite these limitations, the potential of ChatGPT as a supportive tool in ID education and preliminary analysis is evident. However, it should not replace expert consultation, especially in complex clinical decision-making.</p>","PeriodicalId":13600,"journal":{"name":"Infection","volume":" ","pages":"873-881"},"PeriodicalIF":5.4000,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12137519/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Infection","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1007/s15010-024-02350-6","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/12 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"INFECTIOUS DISEASES","Score":null,"Total":0}
引用次数: 0
Abstract
Objectives: Advancements in Artificial Intelligence(AI) have made platforms like ChatGPT increasingly relevant in medicine. This study assesses ChatGPT's utility in addressing bacterial infection-related questions and antibiogram-based clinical cases.
Methods: This study involved a collaborative effort involving infectious disease (ID) specialists and residents. A group of experts formulated six true/false, six open-ended questions, and six clinical cases with antibiograms for four types of infections (endocarditis, pneumonia, intra-abdominal infections, and bloodstream infection) for a total of 96 questions. The questions were submitted to four senior residents and four specialists in ID and inputted into ChatGPT-4 and a trained version of ChatGPT-4. A total of 720 responses were obtained and reviewed by a blinded panel of experts in antibiotic treatments. They evaluated the responses for accuracy and completeness, the ability to identify correct resistance mechanisms from antibiograms, and the appropriateness of antibiotics prescriptions.
Results: No significant difference was noted among the four groups for true/false questions, with approximately 70% correct answers. The trained ChatGPT-4 and ChatGPT-4 offered more accurate and complete answers to the open-ended questions than both the residents and specialists. Regarding the clinical case, we observed a lower accuracy from ChatGPT-4 to recognize the correct resistance mechanism. ChatGPT-4 tended not to prescribe newer antibiotics like cefiderocol or imipenem/cilastatin/relebactam, favoring less recommended options like colistin. Both trained- ChatGPT-4 and ChatGPT-4 recommended longer than necessary treatment periods (p-value = 0.022).
Conclusions: This study highlights ChatGPT's capabilities and limitations in medical decision-making, specifically regarding bacterial infections and antibiogram analysis. While ChatGPT demonstrated proficiency in answering theoretical questions, it did not consistently align with expert decisions in clinical case management. Despite these limitations, the potential of ChatGPT as a supportive tool in ID education and preliminary analysis is evident. However, it should not replace expert consultation, especially in complex clinical decision-making.
期刊介绍：
Infection is a journal dedicated to serving as a global forum for the presentation and discussion of clinically relevant information on infectious diseases. Its primary goal is to engage readers and contributors from various regions around the world in the exchange of knowledge about the etiology, pathogenesis, diagnosis, and treatment of infectious diseases, both in outpatient and inpatient settings.
The journal covers a wide range of topics, including:
Etiology: The study of the causes of infectious diseases.
Pathogenesis: The process by which an infectious agent causes disease.
Diagnosis: The methods and techniques used to identify infectious diseases.
Treatment: The medical interventions and strategies employed to treat infectious diseases.
Public Health: Issues of local, regional, or international significance related to infectious diseases, including prevention, control, and management strategies.
Hospital Epidemiology: The study of the spread of infectious diseases within healthcare settings and the measures to prevent nosocomial infections.
In addition to these, Infection also includes a specialized "Images" section, which focuses on high-quality visual content, such as images, photographs, and microscopic slides, accompanied by brief abstracts. This section is designed to highlight the clinical and diagnostic value of visual aids in the field of infectious diseases, as many conditions present with characteristic clinical signs that can be diagnosed through inspection, and imaging and microscopy are crucial for accurate diagnosis. The journal's comprehensive approach ensures that it remains a valuable resource for healthcare professionals and researchers in the field of infectious diseases.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
