Enhancing Quality of Care Assessments: Managing Atypical Patients in ICU Benchmarking in The Netherlands.

Abstract

Objectives: Benchmarking the quality of care in ICUs contributes to ensuring that patients receive high-quality care. However, the performance metrics in benchmarks may be influenced by atypical patients, those with characteristics that deviate from the typical ICU population. This study aims to provide a framework to identify and reduce the impact of atypical patients in ICU benchmarking, leading to more meaningful ICU performance assessments.

Design: The proposed framework compares patients from each ICU to the aggregated data from the rest of the ICUs to identify patients in the isolated ICU with an atypical clinical data pattern. To benchmark ICU quality of care, we used the standardized mortality ratio (SMR) derived from the Acute Physiology and Chronic Health Evaluation (APACHE)-IV model to evaluate mortality outcomes across ICUs in The Netherlands from 2018 to 2023. We subsequently assessed the impact of excluding these atypical patients on the performance of the APACHE-IV model (expressed as the Brier score) and recalculated the SMRs.

Setting: Three hundred forty-four thousand four hundred fifty-two patients admitted to 75 ICUs across The Netherlands.

Patients: Adult patients admitted to the ICU fulfilling the APACHE-IV inclusion criteria.

Interventions: None.

Measurements and main results: Within the total population, the Brier score of the APACHE-IV prediction model worsened from 0.02 in typical admitted patients to 0.13 in atypical patients. Excluding the top 5% atypical patients from benchmarking analysis altered conclusions for 13 ICUs over the full study period, causing them to move from expected quality range to unexpected quality or vice versa. Furthermore, our analysis identified 6 of the 75 ICUs as admitting the most atypical patients, 5 of which were academic hospitals.

Conclusions: The results highlight how atypical patients negatively affect mortality prediction accuracy and influence ICU quality assessments. Therefore, it is essential to identify and account for these cases when evaluating ICU care by incorporating the proposed framework into routine benchmarking, thereby enhancing the reliability and fairness of performance evaluations.