Journal of Educational Measurement最新文献

Diagnosing Block-Position Effects on the Item Difficulty Scale: A Stage-Specific Localized Difficulty-Shift Model 诊断块位置对项目难度量表的影响：一个特定阶段的局部难度转移模型

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-10-21 DOI: 10.1111/jedm.70069

Tien-Yu Hsieh

{"title":"Diagnosing Block-Position Effects on the Item Difficulty Scale: A Stage-Specific Localized Difficulty-Shift Model","authors":"Tien-Yu Hsieh","doi":"10.1111/jedm.70069","DOIUrl":"https://doi.org/10.1111/jedm.70069","url":null,"abstract":"Block-position effects in large-scale assessments are often modeled as common difficulty shifts, although variation may differ across administration stages and across examinee-item ability-difficulty relationships. This study develops a staged two-parameter logistic framework: M0 has no position effect, M1 adds a common linear shift, M2 permits stage-specific nonlocalized shifts, and M3 allows those shifts to vary with ability-difficulty proximity.The framework was evaluated in controlled balanced and TIMSS-structured sparse simulations and applied to TIMSS 2019 Chinese Taipei Grade 8 mathematics data. Across both simulations, M1 was sufficient for linear effects, M2 for nonlinear but nonlocalized stage effects, and M3 added information when localization was genuinely present. M3 primarily improved recovery of position-effect and probability-effect functions and threshold-proximal calibration, whereas sparse administration reduced precision and weakened identification of the nonlocalized boundary.In the TIMSS application, the common linear trend was small, whereas the empirical administration design separated the items into two disconnected position families, Positions 1 and 4 versus Positions 2 and 3. M3 yielded a proximity-varying fitted pattern aligned with these two families and concentrated near the baseline response threshold. Because no items connected the two families, this between-family pattern cannot be attributed uniquely to block position; the directly connected within-family contrasts did not show clear differences. Global held-out predictive differences were very small. The fitted family-aligned pattern persisted under a weaker item-difficulty prior, although amplitude and localization width were prior-sensitive. The framework therefore supports staged diagnosis of position-related difficulty variation, with M3 used to assess additional localization rather than general predictive superiority.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 4","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149258498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Predicting and Evaluating Item Responses Using Machine Learning, Text-Embeddings, and LLMs 使用机器学习、文本嵌入和llm预测和评估项目反应

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-10-07 DOI: 10.1111/jedm.70067

Evelyn S. Johnson, Hsin-Ro Wei, Onur Demirkaya, Huan Liu, Tong Wu

{"title":"Predicting and Evaluating Item Responses Using Machine Learning, Text-Embeddings, and LLMs","authors":"Evelyn S. Johnson, Hsin-Ro Wei, Onur Demirkaya, Huan Liu, Tong Wu","doi":"10.1111/jedm.70067","DOIUrl":"https://doi.org/10.1111/jedm.70067","url":null,"abstract":"This study evaluates the extent to which synthetic response generation methods reproduce the psychometric properties of Likert-type social-emotional assessment data. We compare four approaches: machine learning (ML), text-embedding models, zero-shot large language models (LLMs), and fine-tuned LLMs, in generating responses to 10 field-test items from the Devereux Student Strengths Assessment (DESSA). Synthetic responses were evaluated using prediction accuracy, graded response model (GRM) parameter recovery, response-style indices, and person-level response patterns. At an aggregate level, ML and text-embedding approaches achieved higher accuracy and more stable recovery of threshold parameters than LLM-based methods. However, closer examination revealed systematic distortions across all methods, including shifts in scale-use behavior and reduced variability in person-level response patterns, particularly for LLM-generated data. These findings suggest that while synthetic methods can approximate surface-level response distributions, they do not consistently reproduce the response processes underlying ordinal data. Implications for the use of synthetic data in early-stage item development and psychometric evaluation are discussed.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 4","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149146385","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Structure-Regularized Composite Scoring for Calibrated MIRT Item Banks 标准MIRT题库的结构正则化综合评分

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-10-04 DOI: 10.1111/jedm.70066

Tien-Yu Hsieh

{"title":"Structure-Regularized Composite Scoring for Calibrated MIRT Item Banks","authors":"Tien-Yu Hsieh","doi":"10.1111/jedm.70066","DOIUrl":"https://doi.org/10.1111/jedm.70066","url":null,"abstract":"Structure-Regularized Composite Scoring (SRCS) was developed as a person-scoring extension for calibrated multidimensional item response theory (MIRT) item banks. Given a Q-matrix, reference distribution, and operational mastery cuts, SRCS combines the calibrated MIRT measurement score with a soft structural score while leaving the item response functions fixed. SRCS was compared with MIRT post hoc classification, MIRT with re-estimated DINA profile scoring (MIRT-reDINA), and Sequential ICC-Induced Diagnostic Scoring (SIDS) across 6 simulation conditions and an empirical illustration using the ECPE grammar data. The first three procedures retained identical conventional MIRT proficiency estimates but differed in their profile-scoring distributions; SRCS alone modified continuous person scoring. In simulation, SRCS incurred small losses in proficiency recovery, classification accuracy, probability calibration, and specificity, while increasing sensitivity and slightly reducing posterior-weighted measurement-structural discrepancy. The soft structural representation was consistently closer to the calibrated MIRT response surface than the hard representation. Empirically, SRCS closely preserved MIRT scores and operational profiles and moderately concentrated profile probabilities, but it did not reduce measurement-structural discrepancy. Overall, SRCS provides a controlled scoring-stage mechanism for incorporating attribute-based structure while preserving a calibrated MIRT measurement system.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 4","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149145603","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Increasing the Efficiency of Comparative Judgment for Writing Assessment: Warm-Starting Estimation and Selection Using NLP 提高写作评价的比较判断效率：用NLP进行热启动估计和选择

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-09-30 DOI: 10.1111/jedm.70065

Michiel De Vrindt, Renske Bouwer, Anaïs Tack, Marije Lesterhuis, Wim Van den Noortgate

{"title":"Increasing the Efficiency of Comparative Judgment for Writing Assessment: Warm-Starting Estimation and Selection Using NLP","authors":"Michiel De Vrindt, Renske Bouwer, Anaïs Tack, Marije Lesterhuis, Wim Van den Noortgate","doi":"10.1111/jedm.70065","DOIUrl":"https://doi.org/10.1111/jedm.70065","url":null,"abstract":"Comparative judgment (CJ) is an assessment method in which assessors compare pairs of essays and judge which is of higher quality. While CJ produces valid and reliable results, it often requires many judgments before quality scores become reliable. In this study, we examine whether natural language processing (NLP) can improve the efficiency of CJ by warm-starting the estimation of quality scores and the selection of essay pairs. We predict essay quality by fine-tuning a transformer model and use these predictions to construct informative priors in a Bayesian Bradley-Terry-Luce model. We then introduce three selection rules based on predicted scores and essay embeddings. In this study, we focus on Dutch writing assessments in secondary and higher education in two settings: one-off assessments and recurrent assessments. Warm-start estimation reduced the number of judgments required to reach a reliability of .70 by approximately 40%-68% compared with likelihood-based cold-start estimation and by approximately 24%-64% compared with Bayesian cold-start estimation. To reach a reliability of .90, warm-start estimation yielded smaller reductions of around 15% compared with cold-start estimation. The warm-start selection rules we constructed yielded smaller additional gains. In practice, a reliability of .70 was reached after two to four judgments per essay, and a reliability of .90 after seven to fourteen judgments. Overall, warm-starting CJ substantially reduces assessor workload and improves the practical applicability of CJ as a summative assessment method.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 4","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149084207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Automated Scoring of Long-Context Essays Using Linear-Complexity Language Models 使用线性复杂性语言模型的长上下文文章自动评分

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-09-28 DOI: 10.1111/jedm.70064

Christopher Ormerod

{"title":"Automated Scoring of Long-Context Essays Using Linear-Complexity Language Models","authors":"Christopher Ormerod","doi":"10.1111/jedm.70064","DOIUrl":"https://doi.org/10.1111/jedm.70064","url":null,"abstract":"Automated essay scoring (AES) increasingly uses pretrained language models, but many operational systems still impose fixed input limits that truncate essays and may weaken score validity. We study how truncation and model architecture affect scoring for long-form writing using more than 20,000 responses. We compare a modern Transformer encoder, ModernBERT, with a linear-time state-space model, Mamba-130M, which supports efficient long-context processing. Evaluation follows the standard framework for the evaluation of automated scoring using Quadratic Weighted Kappa, exact agreement, and standardized mean difference. A paired permutation test (1,000,000 resamples) confirms that truncation produces a statistically significant reduction in QWK for both architectures (<math>\u0000 <semantics>\u0000 <mrow>\u0000 <mi>p</mi>\u0000 <mo><</mo>\u0000 <mo>.</mo>\u0000 <mn>01</mn>\u0000 </mrow>\u0000 <annotation>$p<.01$</annotation>\u0000 </semantics></math>), although the effect size is modest. To assess truncation-related bias, we also examine Kullback-Leibler divergence on propensity score-matched samples and Jeffrey's divergence. Our results show that truncation harms not only scoring accuracy but also construct validity by forcing models to judge incomplete evidence of writing quality. By contrast, the state-space model shows favorable distributional behavior and much lower inference latency as sequence length grows, while maintaining competitive scoring performance. Overall, the findings highlight an important trade-off between efficiency and validity in AES and suggest that linear-time architectures are a promising basis for more faithful evaluation of long-form writing.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-09-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149146675","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Conditional Standard Errors, Posterior Standard Deviations, and Conditional Standard Errors of Prediction in IRT Scoring IRT评分中预测的条件标准误差、后验标准偏差和条件标准误差

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-09-14 DOI: 10.1111/jedm.70059

Peter Baldwin

{"title":"Conditional Standard Errors, Posterior Standard Deviations, and Conditional Standard Errors of Prediction in IRT Scoring","authors":"Peter Baldwin","doi":"10.1111/jedm.70059","DOIUrl":"https://doi.org/10.1111/jedm.70059","url":null,"abstract":"Operational testing programs often rely on different uncertainty summaries to answer different inferential questions. This article distinguishes three such quantities in item response theory (IRT) scoring: the conditional standard error (CSE), which summarizes variability of a reported estimate under repeated measurement at fixed proficiency; the posterior standard deviation (PSD), which summarizes uncertainty about proficiency conditional on observed responses; and the conditional standard error of prediction (CSEP), which summarizes uncertainty in a future reported estimate under specified future administration, scoring, and reporting conditions. The central argument is that uncertainty summaries should be defined by their random quantities, conditioning, and intended uses, with estimation guided by those definitions. CSEP receives particular attention because it is less commonly discussed in operational IRT scoring. Building on classical test theory's standard error of prediction and Bayesian posterior predictive ideas, CSEP is defined as the standard deviation of the posterior predictive distribution of a future reported estimate produced by the operational reporting rule. A posterior predictive Monte Carlo/parametric bootstrap procedure is described for estimating CSEP under arbitrary scoring and reporting rules, with a nested law-of-total-variance extension for decomposing predictive uncertainty into future measurement variability and propagated proficiency uncertainty. An appendix illustrates CSEP estimation and decomposition.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148993988","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Population Difficulty and Discrimination in Item Response Theory: A Complement to Traditional Item Parameters 项目反应理论中的群体困难与歧视：对传统项目参数的补充

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-09-09 DOI: 10.1111/jedm.70061

Joakim Wallmark

引用次数: 0

Variable-Length Calibration Stopping Rules for Multidimensional Computerized Adaptive Testing 多维计算机自适应测试的变长校准停止规则

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-09-02 DOI: 10.1111/jedm.70063

Hengxin Pang, Yuanyuan Qi, Yinhong He, Yiming Zhao

{"title":"Variable-Length Calibration Stopping Rules for Multidimensional Computerized Adaptive Testing","authors":"Hengxin Pang, Yuanyuan Qi, Yinhong He, Yiming Zhao","doi":"10.1111/jedm.70063","DOIUrl":"https://doi.org/10.1111/jedm.70063","url":null,"abstract":"Multidimensional computerized adaptive testing (MCAT) has gained increasing attention, as has online calibration for MCAT. As an essential component of online calibration, the calibration stopping rule determines the conditions for ending calibration and affects its accuracy and efficiency. Current MCAT online calibration studies predominantly employ fixed-length calibration stopping rules. This study introduces four novel variable-length calibration stopping rules (designated as D-rule, E-rule, T-rule, and DI-rule), all derived from the Fisher information matrix of item parameters. The D-rule and E-rule quantify calibration information through the determinant and the minimum eigenvalue of the Fisher information matrix, respectively; the T-rule and DI-rule quantify item parameter estimation error variances using the trace and the maximum diagonal element of the inverse of the Fisher information matrix, respectively. To obtain appropriate termination thresholds, two methods were employed: the empirical method and calibration reliability. Results showed that the four variable-length calibration stopping rules could terminate calibration as expected and achieve satisfactory calibration accuracy.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-09-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148942224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Residual-Based DIF Detection as a Model-Agnostic Framework: A Comparison of Parametric, Semi-Parametric, and Nonparametric Methods 基于残差的DIF检测作为模型不可知的框架：参数、半参数和非参数方法的比较

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-08-31 DOI: 10.1111/jedm.70062

Shan Huang, David Goretzko

{"title":"Residual-Based DIF Detection as a Model-Agnostic Framework: A Comparison of Parametric, Semi-Parametric, and Nonparametric Methods","authors":"Shan Huang, David Goretzko","doi":"10.1111/jedm.70062","DOIUrl":"https://doi.org/10.1111/jedm.70062","url":null,"abstract":"Residual-based methods provide a computationally efficient way to screen items for differential item functioning (DIF) using residuals from a fitted measurement model. Within this broader family, the residual-based differential item functioning (RDIF) statistics represent an established approach that summarizes DIF evidence through group comparisons of raw residuals, squared residuals, and their joint combination. However, the relative behavior of RDIF compared with more flexible residual-based procedures remains unclear, especially under less ideal conditions such as shorter tests, skewed ability distributions, sample imbalance, difficult item profiles, and nonlinear DIF. The present study compares three complementary procedures within a unified residual-based framework: RDIF as a parametric method, spline likelihood-ratio (Spline-LR) as a semi-parametric method, and mutual information (MI) as a nonparametric method. Two simulation studies were conducted in a two-group DIF setting. The first adapted standard RDIF benchmark conditions, whereas the second served as a targeted stress test. Across the simulations, RDIF provided a strong and robust benchmark, showing stable error-control behavior under both conventional and less ideal conditions. Spline-LR and MI showed higher sensitivity under some complex DIF conditions, but these gains were accompanied by substantial losses in specificity and precision. Overall, the findings indicate that greater flexibility in modeling residual dependence does not necessarily improve item-level DIF decisions and that RDIF provides the more reliable basis for routine screening under the conditions examined.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148943749","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Pruned Exact Linear Time Method for Multiple Change-Point Detection in Aberrant Responses 异常响应中多变点检测的精确线性时间剪枝方法

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-08-28 DOI: 10.1111/jedm.70060

Yujie Lu, Yinhong He

{"title":"Pruned Exact Linear Time Method for Multiple Change-Point Detection in Aberrant Responses","authors":"Yujie Lu, Yinhong He","doi":"10.1111/jedm.70060","DOIUrl":"https://doi.org/10.1111/jedm.70060","url":null,"abstract":"In educational measurement, examinees' response patterns often undergo abrupt transitions due to factors such as cheating, random guessing, or test fatigue. These transition points are defined as change points. Accurate identification of aberrant responses is crucial for ensuring test reliability and validity. However, most existing research on aberrant response detection focuses solely on single change-point scenarios. Such methods may be inadequate when examinees exhibit multiple behavioral transitions during a test. This study developed a multiple change-point detection method for response behavior based on the Pruned Exact Linear Time (PELT) algorithm. To address the limitations of fixed penalty criteria in balancing detection power and Type I error control, this study employed an adaptive penalty calibration method based on the false-positive rate. Simulation results showed that the proposed method controlled the Type I error rate at approximately 5% while achieving overall, single-change-point, and multiple-change-point detection rates of 75%, 78%, and 70%, respectively. Empirical analysis of the PISA 2015 financial literacy data revealed that approximately 21% of test-takers exhibited detectable response pattern shifts, with 29% of these test-takers exhibiting multiple change points, demonstrating the method's applicability in real-world low-stakes testing contexts.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-08-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148848921","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0