Educational Measurement-Issues and Practice最新文献_第8页

Bilevel Topic Model-Based Multitask Learning for Constructed-Responses Multidimensional Automated Scoring and Interpretation 基于双层主题模型的多任务学习构建反应多维自动评分和解释

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-03-15 DOI: 10.1111/emip.12550

Jiawei Xiong, Feiming Li

{"title":"Bilevel Topic Model-Based Multitask Learning for Constructed-Responses Multidimensional Automated Scoring and Interpretation","authors":"Jiawei Xiong, Feiming Li","doi":"10.1111/emip.12550","DOIUrl":"10.1111/emip.12550","url":null,"abstract":"Multidimensional scoring evaluates each constructed-response answer from more than one rating dimension and/or trait such as lexicon, organization, and supporting ideas instead of only one holistic score, to help students distinguish between various dimensions of writing quality. In this work, we present a bilevel learning model for combining two objectives, the multidimensional automated scoring, and the students’ writing structure analysis and interpretation. The dual objectives are enabled by a supervised model, called Latent Dirichlet Allocation Multitask Learning (LDAMTL), integrating a topic model and a multitask learning model with an attention mechanism. Two empirical data sets were employed to indicate LDAMTL model performance. On one hand, results suggested that LDAMTL owns better scoring and QW-κ values than two other competitor models, the supervised latent Dirichlet allocation, and Bidirectional Encoder Representations from Transformers at the 5% significance level. On the other hand, extracted topic structures revealed that students with a higher language score tended to employ more compelling words to support the argument in their answers. This study suggested that LDAMTL not only demonstrates the model performance by conjugating the underlying shared representation of each topic and learned representation from the neural networks but also helps understand students’ writing.","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 2","pages":"42-61"},"PeriodicalIF":2.0,"publicationDate":"2023-03-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47037574","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

A Machine Learning Approach for the Simultaneous Detection of Preknowledge in Examinees and Items When Both Are Unknown 一种机器学习方法在未知的情况下同时检测考生和项目中的先验知识

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-24 DOI: 10.1111/emip.12543

Yiqin Pan, James A. Wollack

引用次数: 1

Cheating Detection of Test Collusion: A Study on Machine Learning Techniques and Feature Representation 测试合谋作弊检测:基于机器学习技术和特征表示的研究

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-19 DOI: 10.1111/emip.12538

Shun-Chuan Chang, Keng Lun Chang

引用次数: 1

To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses 评分与否：影响文本回复内容自动评分性能和可行性的因素

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-14 DOI: 10.1111/emip.12544

Torsten Zesch, Andrea Horbach, Fabian Zehner

{"title":"To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses","authors":"Torsten Zesch, Andrea Horbach, Fabian Zehner","doi":"10.1111/emip.12544","DOIUrl":"10.1111/emip.12544","url":null,"abstract":"In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced by other factors such as target population or input modality. Extending previous work, we distinguish conceptual, realization, and nonconformity variance, which are differentially impacted by the various factors. While conceptual variance relates to different concepts embedded in the text responses, realization variance refers to their diverse manifestation through natural language. Nonconformity variance is added by aberrant response behavior. Furthermore, besides its performance, the feasibility of using an automatic scoring system depends on external factors, such as ethical or computational constraints, which influence whether a system with a given performance is accepted by stakeholders. Our work provides (i) a framework for assessment practitioners to decide a priori whether automatic content scoring can be successfully applied in a given setup as well as (ii) new empirical findings and the integration of empirical findings from the literature on factors that influence automatic systems' performance.","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 1","pages":"44-58"},"PeriodicalIF":2.0,"publicationDate":"2023-02-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12544","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44028287","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Call for Papers: Leveraging Measurement for Better Decisions 论文征集：利用衡量做出更好的决策

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-14 DOI: 10.1111/emip.12546

引用次数: 0

Introduction to the Special Section “Issues and Practice in Applying Machine Learning in Educational Measurement” “在教育测量中应用机器学习的问题与实践”专题介绍

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-13 DOI: 10.1111/emip.12547

Zhongmin Cui

引用次数: 0

Machine Learning Literacy for Measurement Professionals: A Practical Tutorial 测量专业人员的机器学习素养：实践教程

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-06 DOI: 10.1111/emip.12539

Rui Nie, Qi Guo, Maxim Morin

引用次数: 1

Causal Inference and COVID: Contrasting Methods for Evaluating Pandemic Impacts Using State Assessments 因果推理与COVID:使用状态评估评估大流行影响的对比方法

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-02-03 DOI: 10.1111/emip.12540

Benjamin R. Shear

{"title":"Causal Inference and COVID: Contrasting Methods for Evaluating Pandemic Impacts Using State Assessments","authors":"Benjamin R. Shear","doi":"10.1111/emip.12540","DOIUrl":"10.1111/emip.12540","url":null,"abstract":"In the spring of 2021, just 1 year after schools were forced to close for COVID-19, state assessments were administered at great expense to provide data about impacts of the pandemic on student learning and to help target resources where they were most needed. Using state assessment data from Colorado, this article describes the biggest threats to making valid inferences about student learning to study pandemic impacts using state assessment data: measurement artifacts affecting the comparability of scores, secular trends, and changes in the tested population. The article compares three statistical approaches (the Fair Trend, baseline student growth percentiles, and multiple regression with demographic covariates) that can support more valid inferences about student learning during the pandemic and in other scenarios in which the tested population changes over time. All three approaches lead to similar inferences about statewide student performance but can lead to very different inferences about student subgroups. Results show that controlling statistically for prepandemic demographic differences can reverse the conclusions about groups most affected by the pandemic and decisions about prioritizing resources.","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 1","pages":"99-109"},"PeriodicalIF":2.0,"publicationDate":"2023-02-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44143301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Machine Learning–Based Profiling in Test Cheating Detection 基于机器学习的测试作弊检测分析

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-01-31 DOI: 10.1111/emip.12541

Huijuan Meng, Ye Ma

引用次数: 0

Psychometric Evaluation of the Preschool Early Numeracy Skills Test–Brief Version Within the Item Response Theory Framework 项目反应理论框架下学前儿童早期算术技能测试的心理测量评价

IF 2 4区教育学

Educational Measurement-Issues and Practice Pub Date : 2023-01-11 DOI: 10.1111/emip.12536

Nikolaos Tsigilis, Katerina Krousorati, Athanasios Gregoriadis, Vasilis Grammatikopoulos

{"title":"Psychometric Evaluation of the Preschool Early Numeracy Skills Test–Brief Version Within the Item Response Theory Framework","authors":"Nikolaos Tsigilis, Katerina Krousorati, Athanasios Gregoriadis, Vasilis Grammatikopoulos","doi":"10.1111/emip.12536","DOIUrl":"10.1111/emip.12536","url":null,"abstract":"The Preschool Early Numeracy Skills Test–Brief Version (PENS-B) is a measure of early numeracy skills, developed and mainly used in the United States. The purpose of this study was to examine the factorial validity and measurement invariance across gender of PENS-B in the Greek educational context. PENS-B was administered to 906 preschool children (473 boys, 433 girls), randomly selected from 84 kindergarten classrooms. A 2PL unidimensional and multidimensional item response theory analysis, using cross-validation procedures, were used to analyze the data. Results showed that responses to 20 items can be adequately explained by a two-dimensional model (Numbering Relations and Arithmetic Operations). Application of differential item functioning procedures did not detect any gender bias. Numeracy Relation comprises 16 items, which assess low levels of this latent trait. On the other hand, four items capture average levels of Arithmetic Operations. Total information curves revealed that both dimensions measure with precision only a small area of their underlying latent trait.","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 2","pages":"32-41"},"PeriodicalIF":2.0,"publicationDate":"2023-01-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12536","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47069473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0