Retrospective validation study of a large language model approach to screening for intraabdominal surgical site infections for quality and safety reporting.

Abstract

Surgical site infections (SSIs), particularly intra-abdominal (IAB) infections, are challenging to identify and remain a resource-intensive focus of infection prevention programmes. Current automated screening measures rely on discrete data from the electronic health record (EHR), such as microbiology results, diagnosis codes and/or return to the operating room. This approach has poor specificity, and therefore surveillance methods depend heavily on additional manual chart review by trained infection preventionists. Large language models (LLMs) offer an opportunity to improve surveillance by synthesising complex clinical documentation alongside structured data elements.We evaluated the performance of a locally hosted LLM (gpt-35-turbo-16k) to improve IAB SSI screening using perioperative clinical notes and microbiology results. The model analysed documentation across the perioperative period (3 days before through 30 days after surgery) to generate case-level SSI summaries and likelihood assessments. We compared the performance of this tool against the current EHR-based screening workflow.Among 1977 abdominal surgical cases, including 56 with confirmed IAB SSIs, the LLM screened 104 cases as high risk, identifying all infections (negative predictive value (NPV) 100%) and achieving a positive predictive value (PPV) of 53.8%. In contrast, the EHR-based workflow identified 288 cases for further review, with a PPV of only 19.4% and the same NPV of 100%. Analysis of 57 224 notes required ~107 million tokens, translating to approximately USD 0.05 per case.An LLM-based approach to SSI surveillance has the potential to substantially improve efficiency while remaining highly accurate and cost-effective. By reducing reliance on manual chart review, this strategy could allow infection preventionists to shift their attention from surveillance toward quality improvement and patient safety initiatives.