Assessing Artificial Intelligence as a Diagnostic Support Tool for Surgical Admissions in the Emergency Department

IF 2.1 3区医学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Journal of Surgical Education Pub Date : 2025-09-01 DOI:10.1016/j.jsurg.2025.103676

Jamie Rice MBBCh, BAO, BA, MSc , Eoin Ó’Briain MBBCh, BAO, BA, MSc , Conor J. Kilkenny MB, BAO, BA, MCh, MRCS , Richard E. Hogan MB, BAO, BA, MCh, MRCS , Tom V. McIntyre MBBCh, BAO, BA, MSc, PGDip, MRCS , Dara Kavanagh MD, FRCSI , Paul C. Neary MD, FRCSI , James M. O’Riordan MD, FRCSI , Shaheel M. Sahebally MD, FRCSI

{"title":"Assessing Artificial Intelligence as a Diagnostic Support Tool for Surgical Admissions in the Emergency Department","authors":"Jamie Rice MBBCh, BAO, BA, MSc , Eoin Ó’Briain MBBCh, BAO, BA, MSc , Conor J. Kilkenny MB, BAO, BA, MCh, MRCS , Richard E. Hogan MB, BAO, BA, MCh, MRCS , Tom V. McIntyre MBBCh, BAO, BA, MSc, PGDip, MRCS , Dara Kavanagh MD, FRCSI , Paul C. Neary MD, FRCSI , James M. O’Riordan MD, FRCSI , Shaheel M. Sahebally MD, FRCSI","doi":"10.1016/j.jsurg.2025.103676","DOIUrl":null,"url":null,"abstract":"<div><h3>INTRODUCTION</h3><div>Artificial intelligence (AI) is increasingly being used in healthcare for data analysis and decision support. This study assesses the potential of AI (specifically ChatGPT-4o) as a diagnostic support tool for surgical admissions in the Emergency Department and compared its accuracy with that of on-call surgical trainees.</div></div><div><h3>METHODS</h3><div>This was a single-institution retrospective study conducted in December 2023. Primary outcomes comprised the agreement and accuracy of diagnoses between AI and trainees. The secondary outcome measure was the similarity of treatment plans. Agreement was defined as the percentage of cases in which both AI and the surgical trainee concurred on the diagnosis or specific aspects of the management plan. Accuracy was defined in both groups as the proportion of provisional diagnoses that corresponded with the definitive diagnosis, as confirmed by subsequent imaging and/or clinical documentation.</div></div><div><h3>RESULTS</h3><div>One hundred patients were included in the study. The mean age of presenting cases was 54 (±20), years. Abdominal pain was the most commonly reported symptom in provisional diagnoses (68%). The accuracy of provisional diagnoses compared to CT-confirmed diagnoses was 76% for the AI system and 74% for the surgical trainees (p = 0.744). Substantial agreement between the AI and trainee’s initial diagnosis was observed (κ = 0.73). There were no statistically significant differences between AI & surgical trainee in provisional diagnosis (p = 0.754), decisions to bring patients to the operating theatre (p = 0.540) and antibiotic administration (p = 0.122). The overall agreement rates were 85%, 58% & 74% respectively.</div></div><div><h3>CONCLUSION</h3><div>Commercially-available AI models demonstrate similar diagnostic ability to junior surgical trainees and may serve as a useful decision-support system. These models could be incorporated into electronic record systems as an adjunct to enhance decision-making.</div></div>","PeriodicalId":50033,"journal":{"name":"Journal of Surgical Education","volume":"82 10","pages":"Article 103676"},"PeriodicalIF":2.1000,"publicationDate":"2025-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Surgical Education","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1931720425002570","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

INTRODUCTION

Artificial intelligence (AI) is increasingly being used in healthcare for data analysis and decision support. This study assesses the potential of AI (specifically ChatGPT-4o) as a diagnostic support tool for surgical admissions in the Emergency Department and compared its accuracy with that of on-call surgical trainees.

METHODS

This was a single-institution retrospective study conducted in December 2023. Primary outcomes comprised the agreement and accuracy of diagnoses between AI and trainees. The secondary outcome measure was the similarity of treatment plans. Agreement was defined as the percentage of cases in which both AI and the surgical trainee concurred on the diagnosis or specific aspects of the management plan. Accuracy was defined in both groups as the proportion of provisional diagnoses that corresponded with the definitive diagnosis, as confirmed by subsequent imaging and/or clinical documentation.

RESULTS

One hundred patients were included in the study. The mean age of presenting cases was 54 (±20), years. Abdominal pain was the most commonly reported symptom in provisional diagnoses (68%). The accuracy of provisional diagnoses compared to CT-confirmed diagnoses was 76% for the AI system and 74% for the surgical trainees (p = 0.744). Substantial agreement between the AI and trainee’s initial diagnosis was observed (κ = 0.73). There were no statistically significant differences between AI & surgical trainee in provisional diagnosis (p = 0.754), decisions to bring patients to the operating theatre (p = 0.540) and antibiotic administration (p = 0.122). The overall agreement rates were 85%, 58% & 74% respectively.

CONCLUSION

Commercially-available AI models demonstrate similar diagnostic ability to junior surgical trainees and may serve as a useful decision-support system. These models could be incorporated into electronic record systems as an adjunct to enhance decision-making.

查看原文本刊更多论文

评估人工智能作为急诊科外科入院的诊断支持工具

人工智能（AI）越来越多地用于医疗保健领域的数据分析和决策支持。本研究评估了人工智能（特别是chatgpt - 40）作为急诊科外科入院诊断支持工具的潜力，并将其准确性与随叫随到的外科实习生进行了比较。方法本研究为单机构回顾性研究，于2023年12月开展。主要结果包括人工智能和受训者之间诊断的一致性和准确性。次要结局指标是治疗方案的相似性。一致性定义为人工智能和外科培训生对诊断或管理计划的特定方面达成一致的病例百分比。准确性在两组中定义为临时诊断与最终诊断相对应的比例，并由随后的影像学和/或临床文件证实。结果100例患者纳入研究。患者平均年龄54（±20）岁。在临时诊断中，腹痛是最常见的症状（68%）。与ct确诊诊断相比，人工智能系统的临时诊断准确率为76%，外科实习生的准确率为74% （p = 0.744）。人工智能与受训者的初始诊断基本一致（κ = 0.73）。人工智能和外科实习生在临时诊断（p = 0.754）、决定将患者带到手术室（p = 0.540）和抗生素给药（p = 0.122）方面无统计学差异。总体同意率分别为85%、58%和74%。结论商用人工智能模型对初级外科培训生具有相似的诊断能力，可作为有用的决策支持系统。这些模式可以纳入电子记录系统，作为加强决策的辅助手段。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Surgical Education EDUCATION, SCIENTIFIC DISCIPLINES-SURGERY

CiteScore

5.60

自引率

10.30%

发文量

261

审稿时长

48 days

期刊介绍： The Journal of Surgical Education (JSE) is dedicated to advancing the field of surgical education through original research. The journal publishes research articles in all surgical disciplines on topics relative to the education of surgical students, residents, and fellows, as well as practicing surgeons. Our readers look to JSE for timely, innovative research findings from the international surgical education community. As the official journal of the Association of Program Directors in Surgery (APDS), JSE publishes the proceedings of the annual APDS meeting held during Surgery Education Week.