Adding New Components to a Composite Quality Metric: How Good Is Good Enough?

Abstract

Objectives: This study illustrates how the statistical reliability of an individual measure relates to the overall reliability of a composite metric, as understanding this relationship provides additional information when evaluating measures for endorsement.

Background: National quality measure endorsement processes typically evaluate individual metrics on criteria such as importance and scientific acceptability (eg, reliability). In practice, quality measures may be used in composite rating systems, which aid in the interpretation of overall quality differences.

Methods: We define an individual measure's reliability by its intraclass correlation and analytically establish the relationship between a composite's reliability and the reliability of its components. We use real data to confirm this relationship under various scenarios. We are motivated by 8 quality measures, which comprise the Quality of Patient Care Star Ratings on Dialysis Facility Care Compare. These measure 4 primary outcomes (mortality, hospitalizations, readmissions, and blood transfusions), vascular access (2 measures), and facility processes (2 measures).

Results: Depending on the reliability of the individual measures, their respective weights in the composite, and their pairwise correlations, there are circumstances when adding a new measure, even if it is less reliable, increases the composite's reliability. For the dialysis facility Star Ratings, we find that the combined reliability of measures grouped within certain domains of care exceeded the reliability of the individual measures within those domains.

Conclusions: New quality measures may add utility to a composite rating system under certain circumstances-a consideration that should, in part, factor into quality measure endorsement processes.