Journal of Educational Measurement最新文献_第2页

KSA Profiles for Standard Setting: A Generative AI Approach to Empirically Grounded Cut Score Evaluation 标准设置的KSA概况：一种基于经验的Cut Score评估的生成AI方法

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-08-21 DOI: 10.1111/jedm.70058

Joseph H. Grochowalski, Claudia Ventura, Merve Sarac, Amy Hendrickson

{"title":"KSA Profiles for Standard Setting: A Generative AI Approach to Empirically Grounded Cut Score Evaluation","authors":"Joseph H. Grochowalski, Claudia Ventura, Merve Sarac, Amy Hendrickson","doi":"10.1111/jedm.70058","DOIUrl":"https://doi.org/10.1111/jedm.70058","url":null,"abstract":"Standard setting for essay-based assessments often relies on expert judgment and performance-level descriptors, but these sources do not always show how examinees at adjacent score levels differ in the knowledge, skills, and abilities (KSAs) they demonstrate. This study introduces a method that uses generative AI to fill that gap. A large language model reads student essays and estimates the likelihood that each student demonstrated a set of pre-defined knowledge, skills, and abilities (KSAs). Those estimates are then summarized as KSA profiles—representations of which competencies are characteristically more or less evident within a score region or at a proposed cut score boundary. To demonstrate the method, we apply it to AP U.S. History and AP World History essay data using hypothetical cut scores as an exploratory example. Results show that profiles differed across score levels, that the same broad KSA structure emerged across both subjects, and that AI-derived measures aligned with established indicators of historical reasoning. We discuss how this kind of evidence could help panels evaluate whether a proposed cut score captures the distinctions a policy intends and inform discussion of proposed cut scores or performance expectations considering evidence of the KSAs students demonstrate.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148784740","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Optimizing Operational Cut Scores under Misclassification Costs and Retake Policies Using IRT-Based Conditional Uncertainty 基于irt的条件不确定性优化误分类成本和重拿策略下的操作切割分数

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-08-10 DOI: 10.1111/jedm.70056

Peter Baldwin, Brian E. Clauser

{"title":"Optimizing Operational Cut Scores under Misclassification Costs and Retake Policies Using IRT-Based Conditional Uncertainty","authors":"Peter Baldwin, Brian E. Clauser","doi":"10.1111/jedm.70056","DOIUrl":"https://doi.org/10.1111/jedm.70056","url":null,"abstract":"Cut scores are typically applied to observed or estimated scores that contain measurement error. Even if the criterion cut score itself were error free, error in examinee scores leads to false-negative and false-positive classifications. The relative costs of these classification errors may be asymmetric and may be further affected by retake policies that increase the probability of passing across repeated administrations. We propose a decision-theoretic procedure for selecting an operational cut score that minimizes a weighted expected misclassification loss while respecting the committee-designated criterion cut score. The method is formulated for item response theory (IRT) scales and accommodates one-, two-, or three-parameter models. Rather than relying on a single reliability-based standard error, we represent measurement error through the conditional sampling distribution of the proficiency estimator given true proficiency, approximated via parametric bootstrap. Expected false-negative and false-positive rates are obtained by Monte Carlo integration over a specified target proficiency distribution, and the optimal operational cut is found by numerical minimization over a feasible cut-score set. The key feature is that misclassification probabilities are computed from an approximation to the conditional sampling distribution of the proficiency estimator, rather than from a reliability-based approximation. An empirical illustration using a three-parameter logistic item response theory model fit to response data from a 36-item examination shows that the optimal operational cut increases as false positives are weighted more heavily and as the number of permitted administrations increases. The proposed approach provides a practical, model-based tool for aligning cut-score policy with explicit misclassification costs and retake rules.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-08-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148753187","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Modeling Item Responses to Understand Change over Time and Diagnostic Needs in Tandem 对项目反应进行建模以理解随时间的变化和诊断需求的串联

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-08-03 DOI: 10.1111/jedm.70057

Stephen Tavares, James Soland

{"title":"Modeling Item Responses to Understand Change over Time and Diagnostic Needs in Tandem","authors":"Stephen Tavares, James Soland","doi":"10.1111/jedm.70057","DOIUrl":"https://doi.org/10.1111/jedm.70057","url":null,"abstract":"Achievement tests are often designed to measure student growth over time using vertical scales while also providing diagnostic information on mastery of specific skill areas. However, traditional assessments rarely integrate psychometric models appropriate for both purposes. This study examines an approach combining vertical scaling with Diagnostic Classification Models (DCMs) to provide both overall achievement scores and subtest-level diagnostic classifications to provide growth and diagnostic information simultaneously. We compared four model specifications against traditional unidimensional vertical scaling using empirical data from the Virginia Language and Literacy Screening System with 17,302 kindergarten and 18,339 first-grade students. We evaluated model fit, score comparability, and classification reliability, and added Monte Carlo simulations to assess parameter recovery. Results revealed important tradeoffs between approaches. While no single approach provided an ideal solution, second-order DCMs with appropriate constraints offer practical viability for operational assessment contexts that wish to measure student growth while also providing diagnostic information.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-08-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.70057","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148752678","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

When Cut Scores Impact Human Ratings: An Extended Many-Facet Rasch Model for Category-Specific Rater Severity Shifts 当削减分数影响人类评级：一个扩展的多面Rasch模型的类别特定的评级严重程度的变化

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-07-19 DOI: 10.1111/jedm.70054

Kuan-Yu Jin, Thomas Eckes

{"title":"When Cut Scores Impact Human Ratings: An Extended Many-Facet Rasch Model for Category-Specific Rater Severity Shifts","authors":"Kuan-Yu Jin, Thomas Eckes","doi":"10.1111/jedm.70054","DOIUrl":"https://doi.org/10.1111/jedm.70054","url":null,"abstract":"The many-facet Rasch model (MFRM), widely used to analyze and evaluate rater-mediated assessments, focuses on between-rater differences in overall severity or leniency across facets and rating scale categories. Studying within-rater severity differences, particularly abrupt shifts in a rater's tendency to assign harsh or lenient ratings around specific scale categories, requires an extended modeling approach. Specifically, to focus on local, category-dependent severity effects, we propose the “many-facet Rasch model for category-specific severity shifts” (MFRM-CSS). Building on Bayesian parameter estimation, we demonstrate the model's suitability for examining local severity changes at scale categories with special significance as cut scores. In a simulation study and real-data analysis, we found that (a) differences in category-dependent severity levels affected observed score distributions and passing rates, (b) the MFRM-CSS reliably recovered true overall and local rater severity parameters, (c) ignoring local severity shifts biased overall severity and category threshold estimates, and (d) the MFRM-CSS outperformed the baseline MFRM, which does not account for category-specific severity shifts, in data-model fit when applied to essay rating data. In the real dataset, the greatest impact of local severity was observed at the second threshold, where the pass-fail cut score was set. The discussion highlights the practical implications of applying the MFRM-CSS and suggests directions for future research.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-07-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148615200","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Group Anchoring in Rater-Mediated Assessments with Sparse Designs and Rater Effects 基于稀疏设计和评级效应的评级中介评估中的群体锚定

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-07-05 DOI: 10.1111/jedm.70053

Stefanie A. Wind, Eli Jones, Brian Wesolowski

{"title":"Group Anchoring in Rater-Mediated Assessments with Sparse Designs and Rater Effects","authors":"Stefanie A. Wind, Eli Jones, Brian Wesolowski","doi":"10.1111/jedm.70053","DOIUrl":"https://doi.org/10.1111/jedm.70053","url":null,"abstract":"Sparse rating designs, where it is not possible for all raters to score all students, are often unavoidable in rater-mediated assessment settings. Unfortunately, sparse ratings can lead to imprecise and inaccurate estimates of model parameters, including student ability, and reduce the detection of rater effects. In the context of latent trait models, researchers have proposed group anchoring as a potential post-data-collection solution to sparseness. We used a simulation study and real-data analysis of ratings from 13 raters of 75 high-school student musical performances to explore the impact of group anchoring on parameter estimates when compared with various disconnected and sparse rating designs. We manipulated the conditions of student sample size, rater effect presence, and rating design. Our results showed that student parameter correlations between fully connected estimates and group anchoring estimates were generally comparable to those from partially connected rating designs, especially with larger sample sizes. However, the results were less stable for estimating rater parameters. Although group anchoring may serve as a robust alternative to fully connected designs for student achievement, it may not always be appropriate when evaluating rater severity.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-07-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148389098","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Beyond a Single Coefficient: A Model-Based Framework for Local Predictive Validity along the Score Scale 超越单一系数：基于模型的局部预测效度框架

IF 1.2 4区心理学

Journal of Educational Measurement Pub Date : 2026-07-05 DOI: 10.1111/jedm.70052

Victoria Florenzano Hernández, Eduardo Alarcón-Bustamante, Inés M. Varas, María Inés Godoy

{"title":"Beyond a Single Coefficient: A Model-Based Framework for Local Predictive Validity along the Score Scale","authors":"Victoria Florenzano Hernández, Eduardo Alarcón-Bustamante, Inés M. Varas, María Inés Godoy","doi":"10.1111/jedm.70052","DOIUrl":"https://doi.org/10.1111/jedm.70052","url":null,"abstract":"Most studies on predictive validity summarize the relationship between admission test scores and academic performance using a single coefficient, such as a correlation or a linear regression slope. While informative, such summaries do not reveal how predictive strength varies across the score scale. In this article, we extend the concept of the marginal effect to the context of predictive validity, formalizing it as the local rate of change in expected academic performance given test scores. The outcome considered is the first-year progress rate, a bounded variable between 0 and 1 that often takes values at the extremes. To respect this distributional structure, we implement a zero/one inflated Beta regression model, which simultaneously accounts for point masses at 0 and 1 and the continuous component in between. This framework enables us to examine how predictive capacity evolves across different score ranges, uncovering regions where test scores are more or less informative. By highlighting such heterogeneity, the approach provides richer insights into how admission tests function and offers valuable tools for analyzing predictive capacity and informing admission policies.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 3","pages":""},"PeriodicalIF":1.2,"publicationDate":"2026-07-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148389179","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Challenges and Potential Solutions of Measurement in the Era of Educational Big Data: Introduction to the JEM Special Issue 教育大数据时代测量的挑战与潜在解决方案：JEM特刊导论

IF 1.6 4区心理学

Journal of Educational Measurement Pub Date : 2026-06-08 DOI: 10.1111/jedm.70050

Kadriye Ercikan, Chun Wang

引用次数: 0

Challenges and Potential Solutions of Measurement in the Era of Educational Big Data: Introduction to the JEM Special Issue 教育大数据时代测量的挑战与潜在解决方案：JEM特刊导论

IF 1.6 4区心理学

Journal of Educational Measurement Pub Date : 2026-06-08 DOI: 10.1111/jedm.70050

Kadriye Ercikan, Chun Wang

引用次数: 0

Automated Coding of Communication Data Using LLM: Consistency across Subgroups 使用LLM的通信数据自动编码：跨子组的一致性