External validation of a widely available, localizable sepsis early detection model in the emergency department setting

Study objective

Several algorithms have been developed to assist clinicians in predicting the onset of sepsis. One of the most widely used is the Epic Sepsis Model. The first release of the model (V1) was a logistic regression that suffered from variable results in external validation. The second release (V2) leverages a gradient boosted tree model that can be localized to an individual hospital system. While widely available, V2 has yet to be independently externally validated.

Methods

We conducted a retrospective study comparing the performance of both models in the emergency department setting. Model discrimination was measured via AUC-ROC for both V1 and V2 at the encounter level, before sepsis-3 criteria were met and before there was evidence of clinical recognition of sepsis (identified by antibiotic, culture, or lactate orders).

Results

35,076 encounters were included. 648 (1.8 %) met sepsis-3 criteria. AUC-ROC scores were 0.77 for V1 and 0.90 for V2. When only considering scores before evidence of clinical recognition of sepsis, there is a drop in AUC-ROC to 0.70 for V1 and to 0.85 for V2. At a scoring threshold targeting a 60 % sensitivity, V1 and V2 predictions were earlier than the first clinical recognition of sepsis in only 33.0 and 33.5 % of cases respectively.

Conclusion

While V2 achieves superior AUC-ROC's to V1 both before and after clinical recognition of sepsis, both models tended to alert for sepsis after evidence of clinical recognition.