External Validation of Persistent Severe Acute Kidney Injury Prediction With Machine Learning Model

Abstract

Objective

To externally validate the persistent electronic alert (PersEA) machine learning model for predicting persistent severe acute kidney injury (psAKI), addressing the scarcity of validated prediction models.

Patients and Methods

We included adult patients (18 years or older) admitted to intensive care unit with at least stage 2 acute kidney injury (AKI) at a tertiary medical center, using retrospective data collected between January 1st, 2017 and December 31st, 2022. The data were accessed and analyzed during the period from March 1st, 2023, through July 28th, 2023. The psAKI was defined as AKI stage 3 lasting for ≥72 hours or AKI leading to death in 48 hours or kidney replacement therapy in 1 day. The performance of the PersEA model, a boosted tree algorithm fed by hourly patient data via electronic health records to provide real-time psAKI predictions, was evaluated using specific metrics that penalize late alarms. We measured the area under the receiver operating characteristic and the area under the precision-recall curves.

Results

After screening, 4479 patients from the Mayo Clinic cohort were included in the current external validation study, with 234 (5.22%) having psAKI. Results from the Amsterdam UMCdb (531 patients, 59 [11.11%] positive) and MIMIC-III (495 patients, 57 [11.52%] positive) cohorts were obtained in a prior development study. The model demonstrated an area under the receiver operating characteristic curve of 0.98 (95% CI, 0.97-0.98) and an area under the precision-recall curve of 0.67 (95% CI, 0.60-0.73), and when applying the threshold that reached 0.80 sensitivity on the internal cohort, PersEA achieved 0.88 sensitivity, 0.94 specificity, and 0.47 precision, all based on Mayo Clinic data.

Conclusion

The PersEA model performed excellently on an external cohort, showing that it is scalable on high-quality data with little to no tuning once a noisy training set is chosen.