Zero-shot large language model application for surgical site infection auditing

Introduction

Artificial intelligence, in particular large language models (LLM), may be able to assist with monitoring for surgical site infections (SSI).

Method

This retrospective study involved the application of the Llama 3.0 70-billion parameter model to the identification of SSI in a group of all SSI in two metropolitan hospitals from a 4-month period. Randomly selected control patients were chosen as comparators. Clinical inpatient and outpatient progress notes were provided to the LLM individually and classified as indicating an SSI or not. These classifications were then analysed to determine binary performance characteristics and to determine the timing of positive case classification.

Results

There was a total of 28 cases in the study, 14 in the case (SSI) group and 14 in the control group. The operations involved in the SSI cases were caesarean section (12/14, 85.7 %) and arthroplasty (2/14, 14.2 %). The LLM had an overall accuracy at the patient-level of 26/28 (93 %). There was a sensitivity of 100 % and specificity of 86%. At the note-level, for the first note flagged by the LLM for each case, 13/14 (92.3 %) were on the same day as, or before, the date noted as the onset of infection as identified by infection control clinicians.

Conclusions

The use of LLM for the screening of medical notes for SSI is feasible. Further studies may seek to evaluate the outcomes of LLM when deployed as part of a clinical workflow.