Re-Evaluating Predictive Performance in the Clinical Risk Analysis Index of Frailty.

Abstract

Objective: To examine whether the Risk Analysis Index (RAI), last calibrated in 2010-2014, requires scoring system changes to remain an effective measure of surgical frailty.

Summary background data: The RAI is among the most validated and widely used measures of surgical frailty. However, like any clinical prediction model, the RAI's scoring system might be prone to algorithmic drift that could invalidate continued use.

Methods: Retrospective cohort study of patients evaluated in outpatient surgery clinics at Veterans Health Administration (VHA) or University of Pittsburgh Medical Center (UPMC). All-cause mortality was measured 180 days after frailty assessment using the RAI-clinical questionnaire (RAI-C).

Results: In 410,715 patients from UPMC and 190,044 patients from VHA, the existing RAI-C scoring system demonstrated strong discrimination (c = 0.868 95% CI [0.864-0.872] for UPMC, 0.827 [0.822-0.832] for VHA) which improved modestly after recalibrating parameter weights (c = 0.872 95% CI [0.868-0.876], for UPMC, 0.839 [0.834-0.845] for VHA). Calibration improved slightly, especially at the extremes where the model tends to underpredict mortality, but little change was observed in the range around the threshold of frailty. Of note, the scoring weights associated with a cancer diagnosis demonstrated some multicollinearity-driven instability, but cancer remained the largest contributor to total score. Decision curve analysis demonstrated that reclassification of just a single additional patient as frail would require screening 1400-6000 patients with the recalibrated system, depending on the threshold score defining frailty, and after accounting for the harms of false-positive classification, weighted proportional to the odds at each threshold probability.

Conclusions: The RAI scoring system remains robust after a decade of widespread use without evidence of meaningful algorithmic drift. The minimal incremental improvement associated with recalibration does not justify the resources required to disseminate and implement a revised scoring system.