Applied Measurement in Education最新文献_第10页

Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring 评分者行为的预测模型：对论文评分质量保证的启示

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-07-02 DOI: 10.1080/08957347.2020.1750406

I. Bejar, Chen Li, D. McCaffrey

{"title":"Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring","authors":"I. Bejar, Chen Li, D. McCaffrey","doi":"10.1080/08957347.2020.1750406","DOIUrl":"https://doi.org/10.1080/08957347.2020.1750406","url":null,"abstract":"ABSTRACT We evaluate the feasibility of developing predictive models of rater behavior, that is, rater-specific models for predicting the scores produced by a rater under operational conditions. In the present study, the dependent variable is the score assigned to essays by a rater, and the predictors are linguistic attributes of the essays used by the e-rater® engine. Specifically, for each rater, the linear regression of rater scores on the linguistic attributes is obtained based on data from two consecutive time periods. The regression from each period was cross validated against data from the other period. Raters were characterized in terms of their level of predictability and the importance of the predictors. Results suggest that rater models capture stable individual differences among raters. To evaluate the feasibility of using rater models as a quality control mechanism, we evaluated the relationship between rater predictability and inter-rater agreement and performance on pre-scored essays. Finally, we conducted a simulation whereby raters are simulated to score exclusively as a function of essay length at different points during the scoring day. We concluded that predictive rater models merit further investigation as a means of quality controlling human scoring.","PeriodicalId":51609,"journal":{"name":"Applied Measurement in Education","volume":"33 1","pages":"234 - 247"},"PeriodicalIF":1.5,"publicationDate":"2020-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/08957347.2020.1750406","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45042742","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Applying Cognitive Theory to the Human Essay Rating Process 认知理论在人类论文评分过程中的应用

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-07-02 DOI: 10.1080/08957347.2020.1750405

B. Finn, Burcu Arslan, M. Walsh

引用次数: 2

Gauging Uncertainty in Test-to-Curriculum Alignment Indices 衡量测试中的不确定性与课程一致性指标

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732387

A. Traynor, Tingxuan Li, Shuqi Zhou

引用次数: 3

The Impact of Test-Taking Disengagement on Item Content Representation 测试脱离对项目内容表示的影响

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732386

S. Wise

引用次数: 12

The Trade-Off between Model Fit, Invariance, and Validity: The Case of PISA Science Assessments 模型拟合、不变性和有效性之间的权衡:以PISA科学评估为例

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732384

Yasmine H. El Masri, D. Andrich

{"title":"The Trade-Off between Model Fit, Invariance, and Validity: The Case of PISA Science Assessments","authors":"Yasmine H. El Masri, D. Andrich","doi":"10.1080/08957347.2020.1732384","DOIUrl":"https://doi.org/10.1080/08957347.2020.1732384","url":null,"abstract":"ABSTRACT In large-scale educational assessments, it is generally required that tests are composed of items that function invariantly across the groups to be compared. Despite efforts to ensure invariance in the item construction phase, for a range of reasons (including the security of items) it is often necessary to account for differential item functioning (DIF) of items post hoc. This typically requires a choice among retaining an item as it is despite its DIF, deleting the item, or resolving (splitting) an item by creating a distinct item for each group. These options involve a trade-off between model fit and the invariance of item parameters, and each option could be valid depending on whether or not the source of DIF is relevant or irrelevant to the variable being assessed. We argue that making a choice requires a careful analysis of statistical DIF and its substantive source. We illustrate our argument by analyzing PISA 2006 science data of three countries (UK, France and Jordan) using the Rasch model, which was the model used for the analyses of all PISA 2006 data. We identify items with real DIF across countries and examine the implications for model fit, invariance, and the validity of cross-country comparisons when these items are either eliminated, resolved or retained.","PeriodicalId":51609,"journal":{"name":"Applied Measurement in Education","volume":"33 1","pages":"174 - 188"},"PeriodicalIF":1.5,"publicationDate":"2020-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/08957347.2020.1732384","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43277464","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 15

Comparing Cut Scores from the Angoff Method and Two Variations of the Hofstee and Beuk Methods 比较Angoff法和Hofstee法和Beuk法的两种变体的切割分数

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732385

Adam E. Wyse

引用次数: 4

Rasch Model Extensions for Enhanced Formative Assessments in MOOCs Rasch模型扩展在mooc中增强形成性评估

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732382

D. Abbakumov, P. Desmet, W. Van den Noortgate

{"title":"Rasch Model Extensions for Enhanced Formative Assessments in MOOCs","authors":"D. Abbakumov, P. Desmet, W. Van den Noortgate","doi":"10.1080/08957347.2020.1732382","DOIUrl":"https://doi.org/10.1080/08957347.2020.1732382","url":null,"abstract":"ABSTRACT Formative assessments are an important component of massive open online courses (MOOCs), online courses with open access and unlimited student participation. Accurate conclusions on students’ proficiency via formative, however, face several challenges: (a) students are typically allowed to make several attempts; and (b) student performance might be affected by other variables, such as interest. Thus, neglecting the effects of attempts and interest in proficiency evaluation might result in biased conclusions. In this study, we try to solve this limitation and propose two extensions of the common psychometric model, the Rasch model, by including the effects of attempts and interest. We illustrate these extensions using real MOOC data and evaluate them using cross-validation. We found that (a) the effects of attempts and interest on the performance are positive on average but both vary among students; (b) a part of the variance in proficiency parameters is due to variation between students in the effect of interest; and (c) the overall accuracy of prediction of student’s item responses using the extensions is 4.3% higher than using the Rasch model.","PeriodicalId":51609,"journal":{"name":"Applied Measurement in Education","volume":"33 1","pages":"113 - 123"},"PeriodicalIF":1.5,"publicationDate":"2020-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/08957347.2020.1732382","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44842343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Subscore Equating and Profile Reporting 分值相等和概要报告

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-03 DOI: 10.1080/08957347.2020.1732381

Euijin Lim, Won‐Chan Lee

引用次数: 4

The Effectiveness and Features of Formative Assessment in US K-12 Education: A Systematic Review 形成性评估在美国K-12教育中的有效性和特点：系统综述

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-03-02 DOI: 10.1080/08957347.2020.1732383

Hansol Lee, Huy Q. Chung, Yu Zhang, J. Abedi, M. Warschauer

{"title":"The Effectiveness and Features of Formative Assessment in US K-12 Education: A Systematic Review","authors":"Hansol Lee, Huy Q. Chung, Yu Zhang, J. Abedi, M. Warschauer","doi":"10.1080/08957347.2020.1732383","DOIUrl":"https://doi.org/10.1080/08957347.2020.1732383","url":null,"abstract":"ABSTRACT In the present article, we present a systematical review of previous empirical studies that conducted formative assessment interventions to improve student learning. Previous meta-analysis research on the overall effects of formative assessment on student learning has been conclusive, but little has been studied on important features of formative assessment interventions and their differential impacts on student learning in the United States’ K-12 education system. Analysis of the identified 126 effect sizes from the selected 33 studies representing 25 research projects that met the inclusion criteria (e.g., included a control condition) revealed an overall small-sized positive effect of formative assessment on student learning (d = .29) with benefits for mathematics (d = .34), literacy (d = .33), and arts (d = .29). Further investigation with meta-regression analyses indicated that supporting student-initiated self-assessment (d = .61) and providing formal formative assessment evidence (e.g., written feedback on quizzes; d = .40) via a medium-cycle length (within or between instructional units; d = .52) were found to enhance the effectiveness of formative assessments.","PeriodicalId":51609,"journal":{"name":"Applied Measurement in Education","volume":"33 1","pages":"124 - 140"},"PeriodicalIF":1.5,"publicationDate":"2020-03-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/08957347.2020.1732383","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42432168","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 44

Some Methods and Evaluation for Linking and Equating with Small Samples 小样本连接与等价的几种方法及评价

IF 1.5 4区教育学

Applied Measurement in Education Pub Date : 2020-01-02 DOI: 10.1080/08957347.2019.1674304

Michael R. Peabody

引用次数: 7