Nicholas Hartman, Vahakn B Shahinian, Valarie B Ashby, Katrina J Price, Kevin He
{"title":"Limitations of the Inter-Unit Reliability: A Set of Practical Examples.","authors":"Nicholas Hartman, Vahakn B Shahinian, Valarie B Ashby, Katrina J Price, Kevin He","doi":"10.1007/s10742-023-00307-0","DOIUrl":null,"url":null,"abstract":"<p><p>Healthcare quality measures are statistics that serve to evaluate healthcare providers and identify those that need to improve their care. Before using these measures in clinical practice, developers and reviewers assess measure reliability, which describes the degree to which differences in the measure values reflect actual variation in healthcare quality, as opposed to random noise. The Inter-Unit Reliability (IUR) is a popular statistic for assessing reliability, and it describes the proportion of total variation in a measure that is attributable to between-provider variation. However, Kalbfleisch, He, Xia, and Li (2018) [<i>Health Services and Outcomes Research Methodology</i>, 18, 215-225] have argued that the IUR has a severe limitation in that some of the between-provider variation may be unrelated to quality of care. In this paper, we illustrate the practical implications of this limitation through several concrete examples. We show that certain best-practices in measure development, such as careful risk adjustment and exclusion of unstable measure values, can decrease the sample IUR value. These findings uncover potential negative consequences of discarding measures with IUR values below some arbitrary threshold.</p>","PeriodicalId":45600,"journal":{"name":"Health Services and Outcomes Research Methodology","volume":null,"pages":null},"PeriodicalIF":1.6000,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11323040/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Health Services and Outcomes Research Methodology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1007/s10742-023-00307-0","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2023/6/28 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0
Abstract
Healthcare quality measures are statistics that serve to evaluate healthcare providers and identify those that need to improve their care. Before using these measures in clinical practice, developers and reviewers assess measure reliability, which describes the degree to which differences in the measure values reflect actual variation in healthcare quality, as opposed to random noise. The Inter-Unit Reliability (IUR) is a popular statistic for assessing reliability, and it describes the proportion of total variation in a measure that is attributable to between-provider variation. However, Kalbfleisch, He, Xia, and Li (2018) [Health Services and Outcomes Research Methodology, 18, 215-225] have argued that the IUR has a severe limitation in that some of the between-provider variation may be unrelated to quality of care. In this paper, we illustrate the practical implications of this limitation through several concrete examples. We show that certain best-practices in measure development, such as careful risk adjustment and exclusion of unstable measure values, can decrease the sample IUR value. These findings uncover potential negative consequences of discarding measures with IUR values below some arbitrary threshold.
期刊介绍:
The journal reflects the multidisciplinary nature of the field of health services and outcomes research. It addresses the needs of multiple, interlocking communities, including methodologists in statistics, econometrics, social and behavioral sciences; designers and analysts of health policy and health services research projects; and health care providers and policy makers who need to properly understand and evaluate the results of published research. The journal strives to enhance the level of methodologic rigor in health services and outcomes research and contributes to the development of methodologic standards in the field. In pursuing its main objective, the journal also provides a meeting ground for researchers from a number of traditional disciplines and fosters the development of new quantitative, qualitative, and mixed methods by statisticians, econometricians, health services researchers, and methodologists in other fields. Health Services and Outcomes Research Methodology publishes: Research papers on quantitative, qualitative, and mixed methods; Case Studies describing applications of quantitative and qualitative methodology in health services and outcomes research; Review Articles synthesizing and popularizing methodologic developments; Tutorials; Articles on computational issues and software reviews; Book reviews; and Notices. Special issues will be devoted to papers presented at important workshops and conferences.