Educational and Psychological Measurement最新文献

{"title":"Recovering Accuracy From Miscalibrated Item Banks: Scoring Error Is Fixable, Selection Error Is Not.","authors":"Xingyao Xiao, Benjamin W Domingue, Nilàm Ram, Michael C Frank","doi":"10.1177/00131644261486854","DOIUrl":"10.1177/00131644261486854","url":null,"abstract":"<p><p>Computerized adaptive testing (CAT) depends on accurately calibrated item banks, but in practice, the item parameters governing deployment of the item banks are often based on small pilot samples, transported across populations, or assigned placeholder values pending proper calibration. Throughout, miscalibration is modeled in the item difficulty parameters, with discriminations held at their true values. We decompose the impact of miscalibration on subsequent ability estimates into two components: item selection error and scoring error. While selection error is not recoverable, rescoring using better calibrations, and without any additional testing burden, can potentially provide a post hoc fix to scoring errors. In a simulation study, we find that rescoring substantially reduces root mean squared error, with scoring error accounting for 86% to 96% of miscalibration-attributable error. In Study 2, we apply this framework to data from 2,657 children (ages 5-12) assessed across three pilot sites in the LEVANTE project. Using identical scoring methods with old versus new item parameters, we found that rescoring shifted scores substantially and reordered children's relative standing; external validation confirms the rescored estimates are better, not just different. Low-ability and younger children are disproportionately affected by rescoring, underscoring the equity implications. These results highlight the utility of rescoring for any CAT using early-stage, transported, or AI-generated item parameters, and how rescoring can enable rapid deployment of assessments to new populations.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261486854"},"PeriodicalIF":4.4,"publicationDate":"2026-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13675416/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149215159","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Sensitivity of Global IRT Fit Indices to the Magnitude and Prevalence of Model-Data Discrepancies.","authors":"Xinyu Liu, David J Weiss","doi":"10.1177/00131644261489511","DOIUrl":"10.1177/00131644261489511","url":null,"abstract":"<p><p>This study examined the operating characteristics of global fit indices for dichotomous item response theory (IRT) models, including the limited-information <math> <mrow> <msub><mrow><mi>M</mi></mrow> <mrow><mn>2</mn></mrow> </msub> </mrow> </math> statistic, root mean square error of approximation (RMSEA), standardized root mean squared residual (SRMSR), comparative fit index (CFI), and Tucker-Lewis index (TLI). A simulation varied sample size, global and localized three-parameter logistic (3PL)-to-two-parameter logistic misspecification, the magnitude and prevalence of surface- and pair-specific underlying local dependence (ULD), null-independence responses, and reversed items under free- or positive-slope estimation. The design included 61 conditions with 1,000 replications per condition, supplemented by large-sample reference analyses. <math> <mrow> <msub><mrow><mi>M</mi></mrow> <mrow><mn>2</mn></mrow> </msub> </mrow> </math> rejection rates were reasonably close to nominal levels under correct specification and increased with stronger and more prevalent local dependence, but sensitivity remained limited for localized 3PL misspecification and weak ULD. RMSEA remained small across most conditions, SRMSR showed substantial sample-size dependence, and CFI and TLI were more responsive to broad or severe discrepancies than to localized violations. Reversed items produced little global discrepancy when negative discriminations were allowed but substantial discrepancy under positive-slope constraints. Overall, global fit indices are best interpreted as complementary screening measures and should be supplemented with targeted diagnostics and parameter inspection.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261489511"},"PeriodicalIF":4.4,"publicationDate":"2026-10-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13669251/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149215084","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Quantifying Sampling Variability of IRT-Based Classification Accuracy and Consistency.","authors":"Youngjin Han, Juyoung Jung","doi":"10.1177/00131644261478043","DOIUrl":"10.1177/00131644261478043","url":null,"abstract":"<p><p>In high-stakes testing, classification accuracy (CA) and classification consistency (CC) provide critical validity evidence for criterion-referenced decisions. However, these indices are conventionally reported as point estimates, obscuring their statistical uncertainty. This study focuses on quantifying the uncertainty associated with item response theory-based classification indices. Because these indices are functions of estimated item parameters, they are subject to sampling variability from parameter estimation. We derive analytical expressions for the asymptotic standard errors of conditional and marginal CA and CC indices under the two-parameter logistic model. In addition, a parameter-drawing multiple imputation approach is implemented as an alternative method for propagating parameter uncertainty. A Monte Carlo simulation study varying sample size and test length evaluates the empirical performance of both estimators. Results indicate that the proposed standard errors closely approximate empirical standard deviations and yield confidence intervals near nominal coverage rates.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261478043"},"PeriodicalIF":4.4,"publicationDate":"2026-10-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13661819/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149180029","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Real-Time Protection of Person-Parameter Updating and Item Selection in Computerized Adaptive Testing Under Item Preknowledge Using Response-Time Evidence.","authors":"Chunyu Piao, Jiwei Zhang, Jing Lu","doi":"10.1177/00131644261489703","DOIUrl":"https://doi.org/10.1177/00131644261489703","url":null,"abstract":"<p><p>Computerized adaptive testing (CAT) is designed to enhance measurement precision and testing efficiency by sequentially updating interim person-parameter estimates and using these estimates to guide subsequent item selection and test termination. However, responses affected by item preknowledge can distort these estimates and, in turn, alter the subsequent course of the adaptive test. We propose a response-time-informed real-time adaptive protection framework that integrates within-examinee monitoring, protected person-parameter updating, and adaptive item selection within the CAT loop. Response time is monitored item by item as a dynamic behavioral indicator using a sequential Bayesian changepoint detection algorithm. When posterior evidence suggests an abrupt within-test change, the response-time signal is combined with response correctness to determine whether the current response is potentially affected by item preknowledge. Flagged responses are excluded from interim ability and speed updating, and the most recent protected estimates guide subsequent item selection and stopping decisions; otherwise, CAT proceeds according to the standard updating rule. We evaluate the framework under alternative item-selection criteria, varying prevalence and severity of item preknowledge, four item time-intensity distributions, and fixed-length and precision-based stopping rules. Simulation results show that the proposed framework achieves strong detection performance at both the examinee and response levels, limits the propagation of contaminated responses through the adaptive loop, and improves the accuracy and precision of person-parameter estimation, particularly when item preknowledge is more prevalent or severe.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261489703"},"PeriodicalIF":4.4,"publicationDate":"2026-10-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13652985/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149165675","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessing Threshold Ordering and Advancement in the Polytomous Rasch Model: A Probability-Based Framework.","authors":"Pasquale Anselmi, Daiana Colledani, Egidio Robusto, Jeremy Hobart, David Andrich","doi":"10.1177/00131644261480297","DOIUrl":"https://doi.org/10.1177/00131644261480297","url":null,"abstract":"<p><p>Rating scales are widely used to measure attitudes, personality traits, and other psychological constructs, yet their response categories may not function as intended. This paper introduces a probability-based framework for interpreting threshold ordering and threshold advancement in the polytomous Rasch model. Specifically, the framework clarifies why threshold ordering and threshold advancement by a useful amount matter for measurement and what is at stake when either requirement is violated. Within this framework, the paper proposes a probability-ratio criterion that links threshold distance to the relative probabilities of adjacent and nonadjacent response categories. The criterion provides researchers and practitioners with a simple, readily interpretable, and context-sensitive basis for determining the minimum threshold distance required to distinguish response categories with a desired probability contrast. The framework is illustrated theoretically and evaluated empirically using the rating scale model and the partial credit model, two parameterisations of the polytomous Rasch model, across a large set of instruments that employ common 4- and 5-point agreement and frequency scales. The empirical findings show that the same rating scale may function differently across instruments, even when administered to the same respondents, and that the central neutral response category in agreement scales warrants particular attention because it frequently exhibits problematic threshold functioning. Taken together, these methodological and empirical contributions advance the analysis of rating-scale functioning using the polytomous Rasch model.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261480297"},"PeriodicalIF":4.4,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13627464/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149105636","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Unordered Polytomous Covariates in Rasch Trees for DIF Detection.","authors":"Andrew T Krist, Jujia Li, Stefanie A Wind","doi":"10.1177/00131644261483181","DOIUrl":"10.1177/00131644261483181","url":null,"abstract":"<p><p>The Rasch Tree model uses model-based recursive partitioning to detect differential item functioning (DIF) across combinations of covariates. This intersectional approach is a powerful exploratory tool, but it has limitations when incorporating unordered polytomous covariates (e.g. race), particularly surrounding interpretability and computational efficiency. As a result, many studies restrict Rasch Tree analyses to contain only dichotomous or continuous covariates. Recent work has proposed collapsing polytomous covariates into fewer categories prior to model fitting, but the effects of this approach have not been assessed. The present study evaluates the impact of collapsing polytomous covariates on Rasch Tree performance using both simulations and empirical data. We compare DIF detection accuracy and model complexity under the default partitioning algorithm and an alternative approach based on <i>k</i>-means clustering. Results indicate that clustering reduces sensitivity and overall DIF detection accuracy. In addition, detection accuracy for polytomous covariates remains low even under the default procedure, highlighting the need for improved methods for incorporating such covariates in Rasch Tree models.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261483181"},"PeriodicalIF":4.4,"publicationDate":"2026-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13597965/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149037133","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Comparison of Item Selection Methods and Parameter Estimation Approaches for Online Calibration in Computerized Adaptive Testing.","authors":"Levent Ertuna, Cees A W Glas, Burcu Atar","doi":"10.1177/00131644261478843","DOIUrl":"10.1177/00131644261478843","url":null,"abstract":"<p><p>Online calibration enables continuous replenishment of computerized adaptive testing (CAT) item banks by embedding new pretest items within operational test sessions. However, the relative performance of different calibration components and their interactions have not been fully examined in a unified framework. This study compared three pretest item selection methods (maximum Fisher information [MFI], D-optimal value design, and Bayesian D-optimal design), two parameter estimation methods (a fixed-ability joint maximum likelihood procedure [JML-itpar] and one expectation-maximization cycle [OEM]), three levels of random calibration stage sample size (250, 500, and 1,000), and three levels of calibration sample size per pretest item (250, 500, and 1,000) under the one-parameter logistic and two-parameter logistic item response theory models through a Monte Carlo simulation with 108 conditions and 100 replications per condition. The results revealed a trade-off among item selection methods: MFI yielded the most accurate difficulty parameter estimates, while D-optimal value design and Bayesian D-optimal design produced better discrimination parameter recovery. Bayesian D-optimal design offered the best balance between accuracy and calibration efficiency. For the first time in the standard online calibration context, JML-itpar was systematically evaluated and found to be a viable estimation method. JML-itpar paired with MFI produced lower discrimination parameter root mean squared error than OEM at calibration sample sizes of 500 and 1,000. OEM remained superior for the difficulty parameter. The calibration sample size per pretest item was the most influential factor, with substantial accuracy gains from 250 to 500 responses and diminishing returns beyond 500. The random calibration stage sample size had negligible effects. These findings provide practical guidance for selecting calibration components in operational CAT programs.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261478843"},"PeriodicalIF":4.4,"publicationDate":"2026-08-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13518613/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148839649","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Two-Step Estimation Approach of Latent Class-Item Response Theory Models in the Presence of DIF and Multidimensionality.","authors":"Zsuzsa Bakk, Rosa Fabbricatore","doi":"10.1177/00131644261467757","DOIUrl":"10.1177/00131644261467757","url":null,"abstract":"<p><p>Estimation of Latent class item response theory (LC-IRT) models is typically performed using full-information maximum likelihood, which may be computationally demanding, susceptible to convergence problems, and subject to interpretational confounding. We propose a two-step estimation approach for unidimensional and multidimensional LC-IRT models with covariates and differential item functioning (DIF). In the first step, the measurement model is estimated, including direct covariate effects on item responses when DIF is part of the final measurement specification. In the second step, covariate effects on latent class membership are estimated while treating the measurement parameters as fixed. We evaluate the two-step estimator through simulation studies and identify the conditions under which it yields low bias, particularly when measurement quality and entropy are adequate. We then apply the proposed approach to data on student performance in statistics.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261467757"},"PeriodicalIF":4.4,"publicationDate":"2026-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13498696/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148789335","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Covariate Inclusion and Class Enumeration in Factor Mixture Modeling: A Monte Carlo Simulation Study.","authors":"Sedat Sen","doi":"10.1177/00131644261472475","DOIUrl":"10.1177/00131644261472475","url":null,"abstract":"<p><p>This Monte Carlo study examined how covariate inclusion affects class enumeration in one-factor, two-class factor mixture models (FMMs). Data were generated under two scenarios: a continuous covariate predicted latent class membership only, or predicted both latent class membership and the latent factor. Conditions varied by class separation, sample size, mixing proportion, and covariate-effect magnitude. Unconditional and one-step conditional FMMs were evaluated using information criteria, likelihood-ratio-based tests, entropy, classification accuracy, and parameter coverage. Results showed that class separation was the strongest determinant of performance, with sample size providing secondary benefits. When the covariate predicted class membership only, conditional FMMs generally improved classification accuracy and became more competitive for class enumeration as the covariate effect increased. When the covariate also predicted the latent factor, these advantages weakened and became more criterion-dependent. Across conditions, the Bayesian information criterion and consistent Akaike's information criterion were the most stable enumeration criteria, whereas entropy and likelihood-ratio-based tests were less reliable. The findings indicate that the value of covariate inclusion in FMM depends on the pathway through which the covariate affects the population model.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261472475"},"PeriodicalIF":4.4,"publicationDate":"2026-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13452891/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148701011","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"When Is Hard Class Assignment Defensible? An Uncertainty-Aware Framework for Psychometric Profile Interpretation.","authors":"Xiaohui Chen, Siguang Chen, Chenglin Wang, Christian Sweeney, Richard Bailey, Nadia Samsudin","doi":"10.1177/00131644261469071","DOIUrl":"10.1177/00131644261469071","url":null,"abstract":"<p><p>Researchers using latent profile and latent class analysis (LPA/LCA) commonly assign individuals to their modal class without evaluating whether this simplification distorts reported class sizes, profile means, or high-severity subgroups. Existing classification-quality diagnostics-entropy, average posterior probabilities, and Masyn's odds of correct classification and classification probability-assess how sharply a model separates its classes, not whether hard-assigned summaries diverge from probability-weighted ones. We address this through an empirical benchmark (four-class SCL-90 solution; <i>N</i> = 59,408), a fully crossed simulation (class separation, class balance, and indicator-class discrimination precision; 27 conditions), and a six-index diagnostic framework. Hard-assigned and probability-weighted summaries were interchangeable under favorable conditions but diverged under low class separation, severe imbalance, or low precision-most acutely for the smallest, most extreme class. The framework offers simulation-calibrated thresholds for documenting assignment adequacy in continuous-indicator LPA; extension to categorical LCA requires further validation. Hard assignment is most defensible when its adequacy is documented rather than assumed.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261469071"},"PeriodicalIF":4.4,"publicationDate":"2026-08-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13442946/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148683597","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}