Compliance Evaluation with ChatGPT for Diagnosis and Treatment in Patients Brought to the ED with a Preliminary Diagnosis of Stroke.

Abstract

Objectives: Chat Generative Pre-trained Transformer (ChatGPT) is a natural language processing product developed by OpenAI. Recently, the use of ChatGPT has gained attention in the field of health care, particularly for its potential applications in diagnostic and decision-making support. While its utility is still being explored, it shows promise as a supplementary tool in these contexts. This study aims to evaluate the potential of ChatGPT in making decisions about 'transportation to the stroke center, suspicion of large vessel occlusion and treatment decisions' of patients brought to the emergency department by ambulance with a preliminary diagnosis of stroke.

Methods: All patients with a stroke code who were transferred to the emergency department (ED) of a tertiary care hospital, Ankara Etlik City Hospital, by ambulance between November 1, 2023, and April 30, 2024, during designated stroke team coverage periods were included in the study. Unlike many stroke centers that operate continuously 24/7, our institution follows a structured on-call system, where specialized stroke teams are assigned time slots to provide stroke care. Data were collected from prehospital records, ED notes, and hospital imaging and treatment records. ChatGPT's decisions were compared to gold standard outcomes using Cohen's kappa test, with sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) calculated for each directive.

Results: A total of 512 patients were analyzed, and ChatGPT's decisions were compared with the patients' final diagnoses and treatments. Analysis comparing ChatGPT's decisions to patient outcomes across prehospital stroke suspicion, large vessel occlusion diagnosis, and treatment phases showed significant agreement (p < 0.001, Kappa: 0.540-0.562). While the sensitivity of the diagnosis of stroke was 91%, the NPV was found to be 98% in patients requiring intravenous tissue plasminogen activator and large vessel occlusion, 97% NPV in patients requiring mechanical thrombectomy.

Conclusions: ChatGPT shows promise as a decision-support tool for identifying acute ischemic stroke and determining treatment needs in prehospital and ED settings. However, its reliance on predefined data highlights the need for physician supervision to address clinical complexities and ensure patient safety. Integrating ChatGPT as an adjunct rather than a standalone system can enhance decision-making efficiency while maintaining high-quality care.