Journal of Educational Measurement最新文献

Sequential Reservoir Computing for Log File‐Based Behavior Process Data Analyses 基于日志文件的行为过程数据分析的顺序储层计算

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-09-14 DOI: 10.1111/jedm.12413

Jiawei Xiong, Shiyu Wang, Cheng Tang, Qidi Liu, Rufei Sheng, Bowen Wang, Huan Kuang, Allan S. Cohen, Xinhui Xiong

{"title":"Sequential Reservoir Computing for Log File‐Based Behavior Process Data Analyses","authors":"Jiawei Xiong, Shiyu Wang, Cheng Tang, Qidi Liu, Rufei Sheng, Bowen Wang, Huan Kuang, Allan S. Cohen, Xinhui Xiong","doi":"10.1111/jedm.12413","DOIUrl":"https://doi.org/10.1111/jedm.12413","url":null,"abstract":"The use of process data in assessment has gained attention in recent years as more assessments are administered by computers. Process data, recorded in computer log files, capture the sequence of examinees' response activities, for example, timestamped keystrokes, during the assessment. Traditional measurement methods are often inadequate for handling this type of data. In this paper, we proposed a sequential reservoir method (SRM) based on a reservoir computing model using the echo state network, with the particle swarm optimization and singular value decomposition as optimization. Designed to regularize features from process data through a computational self‐learning algorithm, this method has been evaluated using both simulated and empirical data. Simulation results suggested that, on one hand, the model effectively transforms action sequences into standardized and meaningful features, and on the other hand, these features are instrumental in categorizing latent behavioral groups and predicting latent information. Empirical results further indicate that SRM can predict assessment efficiency. The features extracted by SRM have been verified as related to action sequence lengths through the correlation analysis. This proposed method enhances the extraction and accessibility of meaningful information from process data, presenting an alternative to existing process data technologies.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"16 1","pages":""},"PeriodicalIF":1.3,"publicationDate":"2024-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142253879","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Exploring Latent Constructs through Multimodal Data Analysis 通过多模态数据分析探索潜在结构

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-08-14 DOI: 10.1111/jedm.12412

Shiyu Wang, Shushan Wu, Yinghan Chen, Luyang Fang, Liang Xiao, Feiming Li

{"title":"Exploring Latent Constructs through Multimodal Data Analysis","authors":"Shiyu Wang, Shushan Wu, Yinghan Chen, Luyang Fang, Liang Xiao, Feiming Li","doi":"10.1111/jedm.12412","DOIUrl":"https://doi.org/10.1111/jedm.12412","url":null,"abstract":"This study presents a comprehensive analysis of three types of multimodal data‐response accuracy, response times, and eye‐tracking data‐derived from a computer‐based spatial rotation test. To tackle the complexity of high‐dimensional data analysis challenges, we have developed a methodological framework incorporating various statistical and machine learning methods. The results of our study reveal that hidden state transition probabilities, based on eye‐tracking features, may be contingent on skill mastery estimated from the fluency CDM model. The hidden state trajectory offers additional diagnostic insights into spatial rotation problem‐solving, surpassing the information provided by the fluency CDM alone. Furthermore, the distribution of participants across different hidden states reflects the intricate nature of visualizing objects in each item, adding a nuanced dimension to the characterization of item features. This complements the information obtained from item parameters in the fluency CDM model, which relies on response accuracy and response time. Our findings have the potential to pave the way for the development of new psychometric and statistical models capable of seamlessly integrating various types of multimodal data. This integrated approach promises more meaningful and interpretable results, with implications for advancing the understanding of cognitive processes involved in spatial rotation tests.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"69 1","pages":""},"PeriodicalIF":1.3,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142211178","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Robustness of Item Response Theory Models under the PISA Multistage Adaptive Testing Designs 国际学生评估项目多阶段适应性测试设计下项目反应理论模型的稳健性

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-08-01 DOI: 10.1111/jedm.12409

Hyo Jeong Shin, Christoph König, Frederic Robin, Andreas Frey, Kentaro Yamamoto

{"title":"Robustness of Item Response Theory Models under the PISA Multistage Adaptive Testing Designs","authors":"Hyo Jeong Shin, Christoph König, Frederic Robin, Andreas Frey, Kentaro Yamamoto","doi":"10.1111/jedm.12409","DOIUrl":"https://doi.org/10.1111/jedm.12409","url":null,"abstract":"Many international large‐scale assessments (ILSAs) have switched to multistage adaptive testing (MST) designs to improve measurement efficiency in measuring the skills of the heterogeneous populations around the world. In this context, previous literature has reported the acceptable level of model parameter recovery under the MST designs when the current item response theory (IRT)‐based scaling models are used. However, previous studies have not considered the influence of realistic phenomena commonly observed in ILSA data, such as item‐by‐country interactions, repeated use of MST designs in subsequent cycles, and nonresponse, including omitted and not‐reached items. The purpose of this study is to examine the robustness of current IRT‐based scaling models to these three factors under MST designs, using the Programme for International Student Assessment (PISA) designs as an example. A series of simulation studies show that the IRT scaling models used in the PISA are robust to repeated use of the MST design in a subsequent cycle with fewer items and smaller sample sizes, while item‐by‐country interactions and items not‐reached have negligible to modest effects on model parameter estimation, and omitted responses have the largest effect. The discussion section provides recommendations and implications for future MST designs and scaling models for ILSAs.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"75 1","pages":""},"PeriodicalIF":1.3,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141882915","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Modeling Nonlinear Effects of Person‐by‐Item Covariates in Explanatory Item Response Models: Exploratory Plots and Modeling Using Smooth Functions 在解释性项目反应模型中模拟逐人项目协变量的非线性效应：探索图和使用平滑函数建模

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-07-25 DOI: 10.1111/jedm.12410

Sun‐Joo Cho, Amanda Goodwin, Matthew Naveiras, Paul De Boeck

{"title":"Modeling Nonlinear Effects of Person‐by‐Item Covariates in Explanatory Item Response Models: Exploratory Plots and Modeling Using Smooth Functions","authors":"Sun‐Joo Cho, Amanda Goodwin, Matthew Naveiras, Paul De Boeck","doi":"10.1111/jedm.12410","DOIUrl":"https://doi.org/10.1111/jedm.12410","url":null,"abstract":"Explanatory item response models (EIRMs) have been applied to investigate the effects of person covariates, item covariates, and their interactions in the fields of reading education and psycholinguistics. In practice, it is often assumed that the relationships between the covariates and the logit transformation of item response probability are linear. However, this linearity assumption obscures the differential effects of covariates over their range in the presence of nonlinearity. Therefore, this paper presents exploratory plots that describe the potential nonlinear effects of person and item covariates on binary outcome variables. This paper also illustrates the use of EIRMs with smooth functions to model these nonlinear effects. The smooth functions examined in this study include univariate smooths of continuous person or item covariates, tensor product smooths of continuous person and item covariates, and by‐variable smooths between a continuous person covariate and a binary item covariate. Parameter estimation was performed using the mgcv R package through the maximum penalized likelihood estimation method. In the empirical study, we identified a nonlinear effect of the person‐by‐item covariate interaction and discussed its practical implications. Furthermore, the parameter recovery and the model comparison method and hypothesis testing procedures presented were evaluated via simulation studies under the same conditions observed in the empirical study.","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"16 1","pages":""},"PeriodicalIF":1.3,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141776807","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

On the Choice of Parameters for the Lognormal Model for Response Times: Commentary on Becker et al. (2013) 关于响应时间对数正态模型参数的选择：对贝克尔等人（2013）的评论

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-07-24 DOI: 10.1111/jedm.12411

Wim J. van der Linden

引用次数: 0

Reckase, M.The Psychometrics of Standard Setting: Connecting Policy and Test Scores: First edition published 2023 by CRC Press, 6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487‐2742 Reckase，M.The Psychometrics of Standard Setting：连接政策与考试分数》：第一版于 2023 年由 CRC Press 出版，地址：6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-07-24 DOI: 10.1111/jedm.12407

Daniel Lewis, Sandip Sinharay

引用次数: 0

Using Automated Procedures to Score Educational Essays Written in Three Languages 使用自动化程序为用三种语言撰写的教育论文评分

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-07-23 DOI: 10.1111/jedm.12406

Tahereh Firoozi, Hamid Mohammadi, Mark J. Gierl

引用次数: 0

A Generalized Objective Function for Computer Adaptive Item Selection 计算机自适应项目选择的通用目标函数

IF 1.3 4区心理学

Journal of Educational Measurement Pub Date : 2024-07-02 DOI: 10.1111/jedm.12405

Harold Doran, Testsuhiro Yamada, Ted Diaz, Emre Gonulates, Vanessa Culver

引用次数: 0

Likelihood-Based Estimation of Model-Derived Oral Reading Fluency 基于似然法估计模型得出的口语阅读流利度

IF 1.4 4区心理学

Journal of Educational Measurement Pub Date : 2024-06-22 DOI: 10.1111/jedm.12404

Cornelis Potgieter, Xin Qiao, Akihito Kamata, Yusuf Kara

引用次数: 0

Curvilinearity in the Reference Composite and Practical Implications for Measurement 参考综合数据的曲线性及其对测量的实际影响

IF 1.4 4区心理学

Journal of Educational Measurement Pub Date : 2024-06-05 DOI: 10.1111/jedm.12402

Xiangyi Liao, Daniel M. Bolt, Jee-Seon Kim

{"title":"Curvilinearity in the Reference Composite and Practical Implications for Measurement","authors":"Xiangyi Liao, Daniel M. Bolt, Jee-Seon Kim","doi":"10.1111/jedm.12402","DOIUrl":"10.1111/jedm.12402","url":null,"abstract":"<p>Item difficulty and dimensionality often correlate, implying that unidimensional IRT approximations to multidimensional data (i.e., reference composites) can take a curvilinear form in the multidimensional space. Although this issue has been previously discussed in the context of vertical scaling applications, we illustrate how such a phenomenon can also easily occur within individual tests. Measures of reading proficiency, for example, often use different task types within a single assessment, a feature that may not only lead to multidimensionality, but also an association between item difficulty and dimensionality. Using a latent regression strategy, we demonstrate through simulations and empirical analysis how associations between dimensionality and difficulty yield a nonlinear reference composite where the weights of the underlying dimensions <i>change</i> across the scale continuum according to the difficulties of the items associated with the dimensions. We further show how this form of curvilinearity produces systematic forms of misspecification in traditional unidimensional IRT models (e.g., 2PL) and can be better accommodated by models such as monotone-polynomial or asymmetric IRT models. Simulations and a real-data example from the Early Childhood Longitudinal Study—Kindergarten are provided for demonstration. Some implications for measurement modeling and for understanding the effects of 2PL misspecification on measurement metrics are discussed.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"61 3","pages":"511-541"},"PeriodicalIF":1.4,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12402","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141386190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0