Educational and Psychological Measurement最新文献_第6页

When Cluster-Robust Inferences Fail. 当群集鲁棒推理失败时。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-12-19 DOI: 10.1177/00131644251393203

Francis Huang

{"title":"When Cluster-Robust Inferences Fail.","authors":"Francis Huang","doi":"10.1177/00131644251393203","DOIUrl":"10.1177/00131644251393203","url":null,"abstract":"Although cluster-robust standard errors (CRSEs) are commonly used to account for violations of observations independence found in nested data, an underappreciated issue is that there are several instances when CRSEs can fail to properly maintain the nominally accepted Type I error rate. These situations (e.g., analyzing data with imbalanced cluster sizes) can readily be found in various types of education-related datasets and are important to consider when computing statistical inference tests when using cluster-level predictors. Using a Monte Carlo simulation, we investigated these conditions and tested alternative estimators and degrees of freedom (df) adjustments to assess how well they could ameliorate the issues related to the use of the traditional CRSE (CR1) estimator using both continuous and dichotomous predictors. Findings showed that the bias-reduced linearization estimator (CR2) and the jackknife estimator (CR3) together with df adjustments were generally effective at maintaining Type I error rates for most of the conditions tested. Results also indicated that the CR1 when paired with df based on the effective cluster size was also acceptable. We emphasize the importance of clearly describing the nested data structure as the characteristics of the dataset can influence Type I error rates when using CRSEs.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251393203"},"PeriodicalIF":4.4,"publicationDate":"2025-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12716980/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145803212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

From Linear Geometry to Nonlinear and Information-Geometric Settings in Test Theory: Bregman Projections as a Unifying Framework. 从线性几何到测试理论中的非线性和信息几何设置：作为统一框架的Bregman投影。

IF 2.3 3区心理学

Educational and Psychological Measurement Pub Date : 2025-12-12 DOI: 10.1177/00131644251393483

Bruno D Zumbo

引用次数: 0

Reliability as Projection in Operator-Theoretic Test Theory: Conditional Expectation, Hilbert Space Geometry, and Implications for Psychometric Practice. 信度在算子理论测试理论中的投射：条件期望、希尔伯特空间几何及其对心理测量实践的影响。

IF 2.3 3区心理学

Educational and Psychological Measurement Pub Date : 2025-11-12 DOI: 10.1177/00131644251389891

Bruno D Zumbo

{"title":"Reliability as Projection in Operator-Theoretic Test Theory: Conditional Expectation, Hilbert Space Geometry, and Implications for Psychometric Practice.","authors":"Bruno D Zumbo","doi":"10.1177/00131644251389891","DOIUrl":"10.1177/00131644251389891","url":null,"abstract":"This article reconceptualizes reliability as a theorem derived from the projection geometry of Hilbert space rather than an assumption of classical test theory. Within this framework, the true score is defined as the conditional expectation <math><mrow><mi>E</mi> <mo>(</mo> <mi>X</mi> <mo>∣</mo> <mi>G</mi> <mo>)</mo></mrow> </math> , representing the orthogonal projection of the observed score onto the σ-algebra of the latent variable. Reliability, expressed as <math><mrow><mi>Rel</mi> <mo>(</mo> <mi>X</mi> <mo>)</mo> <mo>=</mo> <mi>Var</mi> <mo>[</mo> <mi>E</mi> <mo>(</mo> <mi>X</mi> <mo>∣</mo> <mi>G</mi> <mo>)</mo> <mo>]</mo> <mo>/</mo> <mi>Var</mi> <mo>(</mo> <mi>X</mi> <mo>)</mo></mrow> </math> , quantifies the efficiency of this projection-the squared cosine between <math><mrow><mi>X</mi> <mspace></mspace></mrow> </math> and its true-score projection. This formulation unifies reliability with regression <math> <mrow> <msup><mrow><mi>R</mi></mrow> <mrow><mn>2</mn></mrow> </msup> </mrow> </math> , factor-analytic communality, and predictive accuracy in stochastic models. The operator-theoretic perspective clarifies that measurement error corresponds to the orthogonal complement of the projection, and reliability reflects the alignment between observed and latent scores. Numerical examples and measure-theoretic proofs illustrate the framework's generality. The approach provides a rigorous mathematical foundation for reliability, connecting psychometric theory with modern statistical and geometric analysis.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251389891"},"PeriodicalIF":2.3,"publicationDate":"2025-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12615236/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145539189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence. 与序数尺度加权不一致的协议Lambda：类别流行率的校正。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-11-08 DOI: 10.1177/00131644251376553

Rashid Saif Almehrizi

{"title":"Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence.","authors":"Rashid Saif Almehrizi","doi":"10.1177/00131644251376553","DOIUrl":"10.1177/00131644251376553","url":null,"abstract":"Weighted inter-rater agreement allows for differentiation between levels of disagreement among rating categories and is especially useful when there is an ordinal relationship between categories. Many existing weighted inter-rater agreement coefficients are either extensions of weighted Kappa or are formulated as Cohen's Kappa-like coefficients. These measures suffer from the same issues as Cohen's Kappa, including sensitivity to the marginal distributions of raters and the effects of category prevalence. They primarily account for the possibility of chance agreement or disagreement. This article introduces a new coefficient, weighted Lambda, which allows for the inclusion of varying weights assigned to disagreements. Unlike traditional methods, this coefficient does not assume random assignment and does not adjust for chance agreement or disagreement. Instead, it modifies the observed percentage of agreement while taking into account the anticipated impact of prevalence-agreement effects. The study also outlines techniques for estimating sampling standard errors, conducting hypothesis tests, and constructing confidence intervals for weighted Lambda. Illustrative numerical examples and Monte Carlo simulations are presented to investigate and compare the performance of the new weighted Lambda with commonly used weighted inter-rater agreement coefficients across various true agreement levels and agreement matrices. Results demonstrate several advantages of the new coefficient in measuring weighted inter-rater agreement.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251376553"},"PeriodicalIF":4.4,"publicationDate":"2025-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12602299/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145502674","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

On the Complex Sources of Differential Item Functioning: A Comparison of Three Methods. 论差异项目功能的复杂来源：三种方法的比较。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-11-07 DOI: 10.1177/00131644251379802

Haeju Lee, Sijia Huang, Dubravka Svetina Valdivia, Ben Schwartzman

{"title":"On the Complex Sources of Differential Item Functioning: A Comparison of Three Methods.","authors":"Haeju Lee, Sijia Huang, Dubravka Svetina Valdivia, Ben Schwartzman","doi":"10.1177/00131644251379802","DOIUrl":"10.1177/00131644251379802","url":null,"abstract":"Differential item functioning (DIF) has been a long-standing problem in educational and psychological measurement. In practice, the source from which DIF originates can be complex in the sense that an item can show DIF on multiple background variables of different types simultaneously. Although a variety of non-item response theory-(IRT)-based and IRT-based DIF detection methods have been introduced, they do not sufficiently address the issue of DIF evaluation when its source is complex. The recently proposed least absolute shrinkage and selection operator (LASSO) regularization method has shown promising results of detecting DIF on multiple background variables. To provide more insight, in this study, we compared three DIF detection methods, including the non-IRT-based logistic regression (LR), the IRT-based likelihood ratio test (LRT), and LASSO regularization, through a comprehensive simulation and an empirical data analysis. We found that when multiple background variables were considered, the Type I error and Power rates of the three methods for identifying DIF items on one of the variables depended on not only the sample size and its DIF magnitude but also on the DIF magnitude of the other background variable and the correlation between them. We presented other findings and discussed the limitations and future research directions in this paper.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251379802"},"PeriodicalIF":4.4,"publicationDate":"2025-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12602301/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145502628","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An Evaluation of the Replicable Factor Analytic Solutions Algorithm for Variable Selection: A Simulation Study. 变量选择的可复制因子解析解算法的评价：仿真研究。

IF 2.3 3区心理学

Educational and Psychological Measurement Pub Date : 2025-11-03 DOI: 10.1177/00131644251377381

Daniel A Sass, Michael A Sanchez

{"title":"An Evaluation of the Replicable Factor Analytic Solutions Algorithm for Variable Selection: A Simulation Study.","authors":"Daniel A Sass, Michael A Sanchez","doi":"10.1177/00131644251377381","DOIUrl":"10.1177/00131644251377381","url":null,"abstract":"Observed variable and factor selection are critical components of factor analysis, particularly when the optimal subset of observed variables and the number of factors are unknown and results cannot be replicated across studies. The Replicable Factor Analytic Solutions (RFAS) algorithm was developed to assess the replicability of factor structures-both in terms of the number of factors and the variables retained-while identifying the \"best\" or most replicable solutions according to predefined criteria. This study evaluated RFAS performance across 54 experimental conditions that varied in model complexity (six-factor models), interfactor correlations (ρ = 0, .30, and .60), and sample sizes (n = 300, 500, and 1000). Under default settings, RFAS generally performed well and demonstrated its utility in producing replicable factor structures. However, performance declined with highly correlated factors, smaller sample sizes, and more complex models. RFAS was also compared to four alternative variable selection methods: Ant Colony Optimization (ACO), Weighted Group Least Absolute Shrinkage and Selection Operator (LASSO), and stepwise procedures based on target Tucker-Lewis Index (TLI) and ΔTLI criteria. Stepwise and LASSO methods were largely ineffective at eliminating problematic variables under the studied conditions. In contrast, both RFAS and ACO successfully removed variables as intended, although the resulting factor structures often differed substantially between the two approaches. As with other variable selection methods, refining algorithmic criteria may be necessary to further enhance model performance.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251377381"},"PeriodicalIF":2.3,"publicationDate":"2025-11-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12583011/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145451151","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Coefficient Lambda for Interrater Agreement Among Multiple Raters: Correction for Category Prevalence. 多重评价者间一致性的系数Lambda：类别流行率的修正。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-11-03 DOI: 10.1177/00131644251380540

Rashid Saif Almehrizi

{"title":"Coefficient Lambda for Interrater Agreement Among Multiple Raters: Correction for Category Prevalence.","authors":"Rashid Saif Almehrizi","doi":"10.1177/00131644251380540","DOIUrl":"10.1177/00131644251380540","url":null,"abstract":"Fleiss's Kappa is an extension of Cohen's Kappa, developed to assess the degree of interrater agreement among multiple raters or methods classifying subjects using categorical scales. Like Cohen's Kappa, it adjusts the observed proportion of agreement to account for agreement expected by chance. However, over time, several paradoxes and interpretative challenges have been identified, largely stemming from the assumption of random chance agreement and the sensitivity of the coefficient to the number of raters. Interpreting Fleiss's Kappa can be particularly difficult due to its dependence on the distribution of categories and prevalence patterns. This paper argues that a portion of the observed agreement may be better explained by the interaction between category prevalence and inherent category characteristics, such as ambiguity, appeal, or social desirability, rather than by chance alone. By shifting away from the assumption of random rater assignment, the paper introduces a novel agreement coefficient that adjusts for the expected agreement by accounting for category prevalence, providing a more accurate measure of interrater reliability in the presence of imbalanced category distributions. It also examines the theoretical justification for this new measure, its interpretability, its standard error, and the robustness of its estimates in simulation and practical applications.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644251380540"},"PeriodicalIF":4.4,"publicationDate":"2025-11-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12583010/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145451160","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Common Persons Design in Score Equating: A Monte Carlo Investigation. 分数相等中的普通人物设计：蒙特卡洛调查。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-10-29 eCollection Date: 2026-04-01 DOI: 10.1177/00131644251380585

Jiayi Liu, Zhehan Jiang, Tianpeng Zheng, Yuting Han, Shicong Feng

{"title":"Common Persons Design in Score Equating: A Monte Carlo Investigation.","authors":"Jiayi Liu, Zhehan Jiang, Tianpeng Zheng, Yuting Han, Shicong Feng","doi":"10.1177/00131644251380585","DOIUrl":"10.1177/00131644251380585","url":null,"abstract":"The Common Persons (CP) equating design offers critical advantages for high-security testing contexts-eliminating anchor item exposure risks while accommodating non-equivalent groups-yet few studies have systematically examined how CP characteristics influence equating accuracy, and the field still lacks clear implementation guidelines. Addressing this gap, this comprehensive Monte Carlo simulation (N = 5,000 examinees per form; 500 replications) evaluates CP equating by manipulating 8 factors: test length, difficulty shift, ability dispersion, correlation between test forms and CP characteristics. Four equating methods (identity, IRT true-score, linear, equipercentile) were compared using normalized RMSE and %Bias. Key findings reveal: (a) when the CP sample size reaches at least 30, CP sample properties exert negligible influence on accuracy, challenging assumptions about distributional representativeness; (b) Test factors dominate outcomes-difficulty shifts ( <math><mrow><mi>Δ</mi> <msub><mrow><mi>δ</mi></mrow> <mrow><mi>XY</mi></mrow> </msub> </mrow> </math> = 1) degrade IRT precision severely (|%Bias| >22% vs. linear/equipercentile's |%Bias| <1.5%), while longer tests reduce NRMSE and wider ability dispersion ( <math> <mrow> <msub><mrow><mi>σ</mi></mrow> <mrow><mi>θ</mi></mrow> </msub> </mrow> </math> = 1) enhances precision through improved person-item targeting; (c) Equipercentile and linear methods demonstrate superior robustness under form differences. We establish minimum operational thresholds: ≥30 CPs covering the score range suffice for precise equating. These results provide an evidence-based framework for CP implementation by systematically examining multiple manipulated factors, resolving security-vs-accuracy tradeoffs in high-stakes equating (e.g., credentialing exams) and enabling novel solutions like synthetic respondents.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"320-344"},"PeriodicalIF":4.4,"publicationDate":"2025-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12571793/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145430563","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Path Analysis With Mixed-Scale Variables: Categorical ML, Least Squares, and Bayesian Estimations. 路径分析与混合尺度变量：分类机器学习，最小二乘法，和贝叶斯估计。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-10-27 eCollection Date: 2026-06-01 DOI: 10.1177/00131644251379773

Xinya Liang, Paula Castro, Chunhua Cao, Wen-Juo Lo

引用次数: 0

Correcting the Variance of Effect Sizes Based on Binary Outcomes for Clustering. 基于二元结果的聚类效应大小方差校正。

IF 4.4 3区心理学

Educational and Psychological Measurement Pub Date : 2025-10-23 eCollection Date: 2026-04-01 DOI: 10.1177/00131644251380777

Larry V Hedges

引用次数: 0