Early Use of a Risk-Adjusted Mechanical Ventilation Digital Quality Measure Bundle in a Large Health System.

Abstract

Objectives: To describe the development, validation, and deployment of a risk-adjusted digital quality measure (dQM) bundle for spontaneous awakening trials (SATs), spontaneous breathing trials (SBTs), and low-tidal volume ventilation (LTVV) as part of a quality improvement (QI) program in a large health system.

Design: Quasi-experimental before-after study.

Setting: Thirty-seven ICUs across 14 hospitals in the United States.

Patients: Mechanically ventilated patients older than 16 years.

Interventions: An available, open-source, hospital mortality model, a new gradient-boosted ICU mortality model, and four new, heterogenous, stacked ensemble predicted duration of mechanical ventilation (DMV) models (one model predicting up to 14 d of ventilation [14-d DMV model] and three multiple classifier models predicting up to 6 d of ventilation) were created. A regularly refreshing dashboard displaying risk-adjusted information was coupled with audit and feedback sessions for ICU leadership beginning in September 2020.

Measurements and main results: Risk model performance was evaluated, as appropriate, with C-statistics, mean se (MSE), concordance correlation coefficients (CCCs), and F1-scores. Across all ICUs, compliance with SBTs improved from 81 to 97%, LTVV 80 to 90%, and SATs 27 to 65%. Both hospital and ICU mortality models had robust performance, with C-statistics of 0.85 (95% CI, 0.84-0.85) and 0.94 (0.93-0.94), respectively. The 14-day DMV model MSE was 0.63 and CCC was 0.97, whereas the multiple classifier DMV models F1-scores ranged from 0.42 to 0.59. Unadjusted DMV was greater post-implementation (4.32 ± 3.99 d) vs. pre-implementation (3.76 ± 3.66 d). Actual vs. predicted ventilator days were stable pre-implementation vs. post-implementation when assessed with the multiple classifier models and decreased in the post-implementation period when assessed with the 14-day model. Risk-adjusted mortality remained stable.

Conclusions: A dQM bundle proved useful for efficiently tracking process measures related to a ventilator management QI program in a large health system, although risk-adjusted information differed depending on model constructs. Future work should focus on developing and validating generalizable and interoperable dQM bundles.