Evaluating methods to detect variation in hospital level mortality and renal insufficiency within an Australian cardiac surgery registry

Background

Although clinical quality registries have been established worldwide to monitor cardiothoracic surgery outcomes through benchmarking to detect underperforming hospitals (outliers) and improve quality of care, the accuracy of such analyses remains unclear. This study aimed to compare and evaluate methods of outlier classification when applied to real-world and simulated data.

Methods

Data relating to isolated coronary artery bypass graft procedures were obtained from the Australian and New Zealand Society of Cardiac and Thoracic Surgeons Cardiac Surgery Database registry. Unadjusted and risk-adjusted operative mortality and new renal insufficiency were the key outcomes evaluated for two timeframes: cumulative (2018–2021) and rolling (2022); additional data were parametrically generated to simulate these datasets. Agreement in outlier flagging was compared between variations of control limit and confidence interval methods when applied to the real data, and the expected accuracy of the methods evaluated using the simulated data.

Results

While outlier flagging was similar between techniques, agreement between different risk-adjustment, timeframes and significance levels were moderate to poor. The expected accuracy of outlier classification also differed between these considerations, with high performance only reached for risk-adjusted outcomes using cumulative data. Of the methods, outliers flagged using exact binomial 95 % control limits had the highest accuracy.

Conclusions

Clinical registries should consider their data parameters before commencing benchmarking to detect underperforming sites. To optimise accuracy of outlier flagging, outcomes should be risk-adjusted, cumulative datasets should be used in the case of low patient volumes and, where possible, outcomes with higher prevalence should be evaluated.