Continuous predicted risks should be retained when deploying clinical prediction models.

Abstract

Objective: Clinical prediction models are used to obtain predicted risks of a diagnosis or future event. While models can produce continuous predicted probabilities, these are often dichotomised or categorised into risk groups using probability thresholds for the sake of operational convenience. Thresholds or risk groups may be required, but discarding the continuous probability is not, and risks throwing away information that can be useful for prioritising patients. The economic value of continuous risk prediction estimates is not well understood.

Study design and setting: We simulated the impact of ranking patients by predicted risks compared to using risk groups alone when faced with resource constraints at varying levels of model discrimination and event prevalence. We evaluated model performance in terms of positive predictive value, sensitivity, and mean rank of true positives under different levels of model calibration. We then applied our findings to a machine learning-based ordinal scoring system using real data from a large tertiary Singaporean emergency department.

Results: Using predicted probabilities to rank patients by predicted risk led to model performance benefits when compared to risk groups alone. The benefits of avoiding outcome dichotomisation increased as model discrimination and outcome prevalence increased, and ranking was robust to poor model calibration. Repeating this analysis on Singaporean emergency department data showed that benefits of ranking were greatest when resource constraints were highest.

Conclusion: Using continuous probabilities to prioritise patients within risk groups shows the potential for economic benefits. Future prediction models should share equations for deriving continuous risk scores, and deployed models should consider using these scores together with clinical judgement for patient prioritisation.