评估ChatGPT-4从结直肠手术后电子健康记录中检测手术部位感染：一项诊断准确性的试点研究

IF 4.7 3区医学 Q1 INFECTIOUS DISEASES

Journal of Infection and Public Health Pub Date : 2025-02-01 DOI:10.1016/j.jiph.2024.102627

Josep M. Badia , Daniel Casanova-Portoles , Estela Membrilla , Carles Rubiés , Miquel Pujol , Joan Sancho

{"title":"评估ChatGPT-4从结直肠手术后电子健康记录中检测手术部位感染：一项诊断准确性的试点研究","authors":"Josep M. Badia , Daniel Casanova-Portoles , Estela Membrilla , Carles Rubiés , Miquel Pujol , Joan Sancho","doi":"10.1016/j.jiph.2024.102627","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>Surveillance of surgical site infection (SSI) relies on manual methods that are time-consuming and prone to subjectivity. This study evaluates the diagnostic accuracy of ChatGPT for detecting SSI from electronic health records after colorectal surgery via comparison with the results of a nationwide surveillance programme.</div></div><div><h3>Methods</h3><div>This pilot, retrospective, multicentre analysis included 122 patients who underwent colorectal surgery. Patient records were reviewed by both manual surveillance and ChatGPT, which was tasked with identifying SSI and categorizing them as superficial, deep, or organ-space infections. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Receiver operating characteristic (ROC) curve analysis determined the model's diagnostic performance.</div></div><div><h3>Results</h3><div>ChatGPT achieved a sensitivity of 100 %, correctly identifying all SSIs detected by manual methods. The specificity was 54 %, indicating the presence of false positives. The PPV was 67 %, and the NPV was 100 %. The area under the ROC curve was 0.77, indicating good overall accuracy for distinguishing between SSI and non-SSI cases. Minor differences in outcomes were observed between colon and rectal surgeries, as well as between the hospitals participating in the study.</div></div><div><h3>Conclusions</h3><div>ChatGPT shows high sensitivity and good overall accuracy for detecting SSI. It appears to be a useful tool for initial screenings and for reducing manual review workload. The moderate specificity suggests a need for further refinement to reduce the rate of false positives. The integration of ChatGPT alongside electronic medical records, antibiotic consumption and imaging data results for real-time analysis may further improve the surveillance of SSI. ClinicalTrials.gov Identifier: NCT06556017.</div></div>","PeriodicalId":16087,"journal":{"name":"Journal of Infection and Public Health","volume":"18 2","pages":"Article 102627"},"PeriodicalIF":4.7000,"publicationDate":"2025-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Evaluation of ChatGPT-4 for the detection of surgical site infections from electronic health records after colorectal surgery: A pilot diagnostic accuracy study\",\"authors\":\"Josep M. Badia , Daniel Casanova-Portoles , Estela Membrilla , Carles Rubiés , Miquel Pujol , Joan Sancho\",\"doi\":\"10.1016/j.jiph.2024.102627\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Background</h3><div>Surveillance of surgical site infection (SSI) relies on manual methods that are time-consuming and prone to subjectivity. This study evaluates the diagnostic accuracy of ChatGPT for detecting SSI from electronic health records after colorectal surgery via comparison with the results of a nationwide surveillance programme.</div></div><div><h3>Methods</h3><div>This pilot, retrospective, multicentre analysis included 122 patients who underwent colorectal surgery. Patient records were reviewed by both manual surveillance and ChatGPT, which was tasked with identifying SSI and categorizing them as superficial, deep, or organ-space infections. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Receiver operating characteristic (ROC) curve analysis determined the model's diagnostic performance.</div></div><div><h3>Results</h3><div>ChatGPT achieved a sensitivity of 100 %, correctly identifying all SSIs detected by manual methods. The specificity was 54 %, indicating the presence of false positives. The PPV was 67 %, and the NPV was 100 %. The area under the ROC curve was 0.77, indicating good overall accuracy for distinguishing between SSI and non-SSI cases. Minor differences in outcomes were observed between colon and rectal surgeries, as well as between the hospitals participating in the study.</div></div><div><h3>Conclusions</h3><div>ChatGPT shows high sensitivity and good overall accuracy for detecting SSI. It appears to be a useful tool for initial screenings and for reducing manual review workload. The moderate specificity suggests a need for further refinement to reduce the rate of false positives. The integration of ChatGPT alongside electronic medical records, antibiotic consumption and imaging data results for real-time analysis may further improve the surveillance of SSI. ClinicalTrials.gov Identifier: NCT06556017.</div></div>\",\"PeriodicalId\":16087,\"journal\":{\"name\":\"Journal of Infection and Public Health\",\"volume\":\"18 2\",\"pages\":\"Article 102627\"},\"PeriodicalIF\":4.7000,\"publicationDate\":\"2025-02-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Infection and Public Health\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1876034124003617\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"INFECTIOUS DISEASES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Infection and Public Health","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1876034124003617","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"INFECTIOUS DISEASES","Score":null,"Total":0}

引用次数: 0

摘要

背景：手术部位感染（SSI）的监测依赖于人工方法，既费时又容易主观。本研究通过与全国监测计划的结果比较，评估了ChatGPT在结直肠手术后从电子健康记录中检测SSI的诊断准确性。方法：这项前瞻性、回顾性、多中心分析包括122例接受结直肠手术的患者。通过人工监测和ChatGPT审查患者记录，ChatGPT的任务是识别SSI并将其分类为浅表、深部或器官间隙感染。计算敏感性、特异性、阳性预测值（PPV）和阴性预测值（NPV）。受试者工作特征（ROC）曲线分析决定了模型的诊断性能。结果：ChatGPT的灵敏度为100 %，能正确识别所有手工方法检测到的ssi。特异性为54 %，提示存在假阳性。PPV为67 %，NPV为100 %。ROC曲线下面积为0.77，表明区分SSI和非SSI病例的总体准确性较好。在结肠和直肠手术之间，以及参与研究的医院之间，观察到结果的微小差异。结论：ChatGPT检测SSI具有较高的灵敏度和较好的整体准确度。它似乎是初始筛选和减少人工审查工作量的有用工具。中等特异性表明需要进一步改进以降低假阳性率。ChatGPT与电子病历、抗生素用量和成像数据结果的整合，可用于实时分析，从而进一步改善对SSI的监测。临床试验：gov标识符：NCT06556017。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Evaluation of ChatGPT-4 for the detection of surgical site infections from electronic health records after colorectal surgery: A pilot diagnostic accuracy study

Background

Surveillance of surgical site infection (SSI) relies on manual methods that are time-consuming and prone to subjectivity. This study evaluates the diagnostic accuracy of ChatGPT for detecting SSI from electronic health records after colorectal surgery via comparison with the results of a nationwide surveillance programme.

Methods

This pilot, retrospective, multicentre analysis included 122 patients who underwent colorectal surgery. Patient records were reviewed by both manual surveillance and ChatGPT, which was tasked with identifying SSI and categorizing them as superficial, deep, or organ-space infections. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Receiver operating characteristic (ROC) curve analysis determined the model's diagnostic performance.

Results

ChatGPT achieved a sensitivity of 100 %, correctly identifying all SSIs detected by manual methods. The specificity was 54 %, indicating the presence of false positives. The PPV was 67 %, and the NPV was 100 %. The area under the ROC curve was 0.77, indicating good overall accuracy for distinguishing between SSI and non-SSI cases. Minor differences in outcomes were observed between colon and rectal surgeries, as well as between the hospitals participating in the study.

Conclusions

ChatGPT shows high sensitivity and good overall accuracy for detecting SSI. It appears to be a useful tool for initial screenings and for reducing manual review workload. The moderate specificity suggests a need for further refinement to reduce the rate of false positives. The integration of ChatGPT alongside electronic medical records, antibiotic consumption and imaging data results for real-time analysis may further improve the surveillance of SSI. ClinicalTrials.gov Identifier: NCT06556017.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Journal of Infection and Public Health PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH -INFECTIOUS DISEASES

CiteScore

13.10

自引率

1.50%

发文量

203

审稿时长

96 days

期刊介绍： The Journal of Infection and Public Health, first official journal of the Saudi Arabian Ministry of National Guard Health Affairs, King Saud Bin Abdulaziz University for Health Sciences and the Saudi Association for Public Health, aims to be the foremost scientific, peer-reviewed journal encompassing infection prevention and control, microbiology, infectious diseases, public health and the application of healthcare epidemiology to the evaluation of health outcomes. The point of view of the journal is that infection and public health are closely intertwined and that advances in one area will have positive consequences on the other. The journal will be useful to all health professionals who are partners in the management of patients with communicable diseases, keeping them up to date. The journal is proud to have an international and diverse editorial board that will assist and facilitate the publication of articles that reflect a global view on infection control and public health, as well as emphasizing our focus on supporting the needs of public health practitioners. It is our aim to improve healthcare by reducing risk of infection and related adverse outcomes by critical review, selection, and dissemination of new and relevant information in the field of infection control, public health and infectious diseases in all healthcare settings and the community.