A methodological systematic review of validation and performance of sepsis real-time prediction models

Abstract

Sepsis real-time prediction models (SRPMs) provide timely alerts and may improve patient outcomes but face limited clinical adoption due to inconsistent validation methods and potential biases. Comprehensive evaluation, including external full-window validation with model- and outcome-level metrics, is crucial for real-world effectiveness, yet performance evidence remains scarce. This study systematically reviewed SRPM performance across validation methods, analyzing 91 studies from multiple databases. Only 54.9% applied full-window validation with both metric types. Performance decreased under external and full-window validation, with median AUROCs of 0.886 and 0.861 at 6- and 12-hours pre-onset, dropping to 0.783 in full-window external validation. Median Utility Scores declined from 0.381 in internal to -0.164 in external validation. Combining AUROC and Utility Score identified top-performing SRPMs in 18.7% of studies. Hand-crafted features significantly improved performance. Future research should focus on multi-center datasets, hand-crafted features, multi-metric full-window validation, and prospective trials to support clinical implementation.