Dynamic updating of psychosis prediction models in individuals at ultra high-risk of psychosis.

Abstract

Background: The performance of psychiatric risk calculators can deteriorate over time due to changes in patient population, referral pathways, and medical advances. Such temporal biases in existing models may lead to suboptimal decisions when translated into clinical practice. Methods are available to correct this bias, yet no research has been conducted to investigate their utility in psychiatry.

Methods: We aimed to analyse the performance of model updating methods for predicting psychosis onset by one year in 784 individuals at ultra high-risk (UHR) of psychosis from the UHR 1000+ cohort - a longitudinal cohort of UHR individuals recruited to research studies at Orygen, Melbourne, Australia, between 1995 and 2020. Model updating was performed using a yearly adjusted model (recalibration), a continuously updated model (refitting), and a continuous Bayesian updating model (dynamic updating) and compared to a static logistic regression prediction model (original) regarding calibration, discrimination, and clinical net benefit.

Results: The original model was poorly calibrated over the entire validation period. All three updating methods improved the predictive performance compared to the original model (recalibration: P= 0.014, refitting: P= 0.028, dynamic updating: P= 0.002). The dynamic updating method demonstrated the best predictive performance (Harrel's C-index = 0.70, 95% CI: [0.58, 0.81]), calibration slope (slope = 1.03, 95% CI: [0.38, 1.74]) and clinical net benefit over the entire validation period.

Conclusions: Dynamic updating of psychosis prediction models may help to mitigate decreases in performance over time. Hence, existing psychosis prediction models need to be monitored for temporal biases to mitigate potentially harmful decisions.