Journal of Educational Measurement最新文献_第7页

Using Simulated Retests to Estimate the Reliability of Diagnostic Assessment Systems 用模拟复验估计诊断评估系统的可靠性

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2023-02-19 DOI: 10.1111/jedm.12359

W. Jake Thompson, Brooke Nash, Amy K. Clark, Jeffrey C. Hoover

{"title":"Using Simulated Retests to Estimate the Reliability of Diagnostic Assessment Systems","authors":"W. Jake Thompson, Brooke Nash, Amy K. Clark, Jeffrey C. Hoover","doi":"10.1111/jedm.12359","DOIUrl":"10.1111/jedm.12359","url":null,"abstract":"As diagnostic classification models become more widely used in large-scale operational assessments, we must give consideration to the methods for estimating and reporting reliability. Researchers must explore alternatives to traditional reliability methods that are consistent with the design, scoring, and reporting levels of diagnostic assessment systems. In this article, we describe and evaluate a method for simulating retests to summarize reliability evidence at multiple reporting levels. We evaluate how the performance of reliability estimates from simulated retests compares to other measures of classification consistency and accuracy for diagnostic assessments that have previously been described in the literature, but which limit the level at which reliability can be reported. Overall, the findings show that reliability estimates from simulated retests are an accurate measure of reliability and are consistent with other measures of reliability for diagnostic assessments. We then apply this method to real data from the Examination for the Certificate of Proficiency in English to demonstrate the method in practice and compare reliability estimates from observed data. Finally, we discuss implications for the field and possible next directions.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"60 3","pages":"455-475"},"PeriodicalIF":1.3,"publicationDate":"2023-02-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47801652","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An Exploration of an Improved Aggregate Student Growth Measure Using Data from Two States 利用两个州的数据探索一种改进的学生综合成长测量方法

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2023-01-31 DOI: 10.1111/jedm.12354

Katherine E. Castellano, Daniel F. McCaffrey, J. R. Lockwood

{"title":"An Exploration of an Improved Aggregate Student Growth Measure Using Data from Two States","authors":"Katherine E. Castellano, Daniel F. McCaffrey, J. R. Lockwood","doi":"10.1111/jedm.12354","DOIUrl":"10.1111/jedm.12354","url":null,"abstract":"The simple average of student growth scores is often used in accountability systems, but it can be problematic for decision making. When computed using a small/moderate number of students, it can be sensitive to the sample, resulting in inaccurate representations of growth of the students, low year-to-year stability, and inequities for low-incidence groups. An alternative designed to address these issues is to use an Empirical Best Linear Prediction (EBLP), which is a weighted average of growth score data from other years and/or subjects. We apply both approaches to two statewide datasets to answer empirical questions about their performance. The EBLP outperforms the simple average in accuracy and cross-year stability with the exception that accuracy was not necessarily improved for very large districts in one of the states. In such exceptions, we show a beneficial alternative may be to use a hybrid approach in which very large districts receive the simple average and all others receive the EBLP. We find that adding more growth score data to the computation of the EBLP can improve accuracy, but not necessarily for larger schools/districts. We review key decision points in aggregate growth reporting and in specifying an EBLP weighted average in practice.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"60 2","pages":"173-201"},"PeriodicalIF":1.3,"publicationDate":"2023-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41556588","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Classification Accuracy and Consistency of Compensatory Composite Test Scores 补偿性综合测试成绩的分类准确性和一致性

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2023-01-28 DOI: 10.1111/jedm.12357

J. Carl Setzer, Ying Cheng, Cheng Liu

引用次数: 0

Editorial for JEM issue 59-4 JEM第59-4期社论

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2023-01-06 DOI: 10.1111/jedm.12356

Sandip Sinharay

引用次数: 0

Specifying the Three Ws in Educational Measurement: Who Uses Which Scores for What Purpose? 指定教育测量中的三个w:谁使用哪个分数用于什么目的?

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-12-25 DOI: 10.1111/jedm.12355

Andrew Ho

引用次数: 1

Online Calibration in Multidimensional Computerized Adaptive Testing with Polytomously Scored Items 多维计算机自适应测试中的在线标定

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-12-15 DOI: 10.1111/jedm.12353

Lu Yuan, Yingshi Huang, Shuhang Li, Ping Chen

引用次数: 0

Measuring the Uncertainty of Imputed Scores 估算分数的不确定度测量

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-12-14 DOI: 10.1111/jedm.12352

Sandip Sinharay

引用次数: 1

An Exponentially Weighted Moving Average Procedure for Detecting Back Random Responding Behavior 一种指数加权移动平均法检测反向随机响应行为

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-12-09 DOI: 10.1111/jedm.12351

Yinhong He

{"title":"An Exponentially Weighted Moving Average Procedure for Detecting Back Random Responding Behavior","authors":"Yinhong He","doi":"10.1111/jedm.12351","DOIUrl":"10.1111/jedm.12351","url":null,"abstract":"Back random responding (BRR) behavior is one of the commonly observed careless response behaviors. Accurately detecting BRR behavior can improve test validities. Yu and Cheng (2019) showed that the change point analysis (CPA) procedure based on weighted residual (CPA-WR) performed well in detecting BRR. Compared with the CPA procedure, the exponentially weighted moving average (EWMA) obtains more detailed information. This study equipped the weighted residual statistic with EWMA, and proposed the EWMA-WR method to detect BRR. To make the critical values adaptive to the ability levels, this study proposed the Monte Carlo simulation with ability stratification (MC-stratification) method for calculating critical values. Compared to the original Monte Carlo simulation (MC) method, the newly proposed MC-stratification method generated a larger number of satisfactory results. The performances of CPA-WR and EWMA-WR were evaluated under different conditions that varied in the test lengths, abnormal proportions, critical values and smoothing constants used in the EWMA-WR method. The results showed that EWMA-WR was more powerful than CPA-WR in detecting BRR. Moreover, an empirical study was conducted to illustrate the utility of EWMA-WR for detecting BRR.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"60 2","pages":"282-317"},"PeriodicalIF":1.3,"publicationDate":"2022-12-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47390314","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Multiple-Group Joint Modeling of Item Responses, Response Times, and Action Counts with the Conway-Maxwell-Poisson Distribution Conway‐Maxwell‐Poisson分布的项目响应、响应时间和行动次数的多组联合建模

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-12-07 DOI: 10.1111/jedm.12349

Xin Qiao, Hong Jiao, Qiwei He

{"title":"Multiple-Group Joint Modeling of Item Responses, Response Times, and Action Counts with the Conway-Maxwell-Poisson Distribution","authors":"Xin Qiao, Hong Jiao, Qiwei He","doi":"10.1111/jedm.12349","DOIUrl":"10.1111/jedm.12349","url":null,"abstract":"Multiple group modeling is one of the methods to address the measurement noninvariance issue. Traditional studies on multiple group modeling have mainly focused on item responses. In computer-based assessments, joint modeling of response times and action counts with item responses helps estimate the latent speed and action levels in addition to latent ability. These two new data sources can also be used to further address the measurement noninvariance issue. One challenge, however, is to correctly model action counts which can be underdispersed, overdispersed, or equidispersed in real data sets. To address this, we adopted the Conway-Maxwell-Poisson distribution that accounts for different types of dispersion in action counts and incorporated it in the multiple group joint modeling of item responses, response times, and action counts. Bayesian Markov Chain Monte Carlo method was used for model parameter estimation. To illustrate an application of the proposed model, an empirical data analysis was conducted using the Programme for International Student Assessment (PISA) 2015 collaborative problem-solving items where potential measurement noninvariance issue existed between gender groups. Results indicated that Conway-Maxwell-Poisson model yielded better model fit than alternative count data models such as negative binomial and Poisson models. In addition, response times and action counts provided further information on performance differences between groups.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"60 2","pages":"255-281"},"PeriodicalIF":1.3,"publicationDate":"2022-12-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45484845","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

NCME Presidential Address 2022: Turning the Page to the Next Chapter of Educational Measurement 全国教育计量学会2022年会长致辞:掀开教育计量的新篇章

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2022-11-09 DOI: 10.1111/jedm.12350

Derek C. Briggs

引用次数: 0