Reliability for multilevel data: A correlation approach.

IF 7.6 1区心理学 Q1 PSYCHOLOGY, MULTIDISCIPLINARY

Psychological methods Pub Date : 2025-05-22 DOI:10.1037/met0000738

Tzu-Yao Lin, Francis Tuerlinckx, Sophie Vanbelle

{"title":"Reliability for multilevel data: A correlation approach.","authors":"Tzu-Yao Lin, Francis Tuerlinckx, Sophie Vanbelle","doi":"10.1037/met0000738","DOIUrl":null,"url":null,"abstract":"<p><p>Studying the reliability of a measurement instrument is essential. Despite the recognition of the importance of reliability in psychology and medicine and the various reliability coefficients that have been proposed, research on reliability for nested or multilevel data, ubiquitously in observational studies, remains limited. Two recent articles (Schönbrodt et al., 2022; ten Hove et al., 2022) address how to quantify reliability in multilevel settings based on generalizability theory. Specifically, ten Hove et al. (2022) defined between-cluster and within-cluster interrater intraclass correlation coefficients for multilevel designs where persons or raters are nested within clusters. Schönbrodt et al. (2022) also defined reliability coefficients at between-cluster and within-cluster (i.e., between-person) levels for designs where persons nested in couples are assessed numerous times daily over a number of days. Nevertheless, when applied to a common design, both approaches give inconsistent results regarding their definition of cluster-level reliability. In this article, we propose an alternative approach to defining reliability coefficients for multilevel data that are based on calculating the expected correlation between repeated measurements. We will compare our approach with that of Schönbrodt et al. (2022) and ten Hove et al. (2022) and explain the differences between the three approaches in a number of common nested data structures: (a) raters crossed with both persons and clusters, but persons are nested within clusters, (b) raters nested within both persons and clusters, and (c) persons nested in clusters and crossed with raters and days. (PsycInfo Database Record (c) 2025 APA, all rights reserved).</p>","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6000,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Psychological methods","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1037/met0000738","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

Studying the reliability of a measurement instrument is essential. Despite the recognition of the importance of reliability in psychology and medicine and the various reliability coefficients that have been proposed, research on reliability for nested or multilevel data, ubiquitously in observational studies, remains limited. Two recent articles (Schönbrodt et al., 2022; ten Hove et al., 2022) address how to quantify reliability in multilevel settings based on generalizability theory. Specifically, ten Hove et al. (2022) defined between-cluster and within-cluster interrater intraclass correlation coefficients for multilevel designs where persons or raters are nested within clusters. Schönbrodt et al. (2022) also defined reliability coefficients at between-cluster and within-cluster (i.e., between-person) levels for designs where persons nested in couples are assessed numerous times daily over a number of days. Nevertheless, when applied to a common design, both approaches give inconsistent results regarding their definition of cluster-level reliability. In this article, we propose an alternative approach to defining reliability coefficients for multilevel data that are based on calculating the expected correlation between repeated measurements. We will compare our approach with that of Schönbrodt et al. (2022) and ten Hove et al. (2022) and explain the differences between the three approaches in a number of common nested data structures: (a) raters crossed with both persons and clusters, but persons are nested within clusters, (b) raters nested within both persons and clusters, and (c) persons nested in clusters and crossed with raters and days. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

查看原文本刊更多论文

多水平数据的可靠性：一种相关方法。

研究测量仪器的可靠性是必要的。尽管人们认识到可靠性在心理学和医学中的重要性，并提出了各种可靠性系数，但在观察性研究中无处不在的嵌套或多层数据的可靠性研究仍然有限。最近的两篇文章(Schönbrodt et al., 2022；（10 Hove等人，2022）解决了如何基于概率性理论在多级设置中量化可靠性。具体而言，10 Hove等人（2022）定义了人员或评分员嵌套在聚类中的多层次设计的聚类间和聚类内评分者类内相关系数。Schönbrodt等人（2022）还定义了集群间和集群内（即人与人之间）水平的可靠性系数，其中嵌套在夫妻中的人在数天内每天进行多次评估。然而，当应用于共同设计时，两种方法给出的关于集群级可靠性定义的结果不一致。在本文中，我们提出了一种替代方法来定义基于计算重复测量之间的期望相关性的多水平数据的可靠性系数。我们将把我们的方法与Schönbrodt等人（2022）和ten Hove等人（2022）的方法进行比较，并解释在许多常见嵌套数据结构中三种方法之间的差异：(a)评分者与人员和集群交叉，但人员嵌套在集群中，(b)评分者嵌套在人员和集群中，以及(c)人员嵌套在集群中并与评分者和天数交叉。（PsycInfo Database Record (c) 2025 APA，版权所有）。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Psychological methods PSYCHOLOGY, MULTIDISCIPLINARY-

CiteScore

13.10

自引率

7.10%

发文量

159

期刊介绍： Psychological Methods is devoted to the development and dissemination of methods for collecting, analyzing, understanding, and interpreting psychological data. Its purpose is the dissemination of innovations in research design, measurement, methodology, and quantitative and qualitative analysis to the psychological community; its further purpose is to promote effective communication about related substantive and methodological issues. The audience is expected to be diverse and to include those who develop new procedures, those who are responsible for undergraduate and graduate training in design, measurement, and statistics, as well as those who employ those procedures in research.