Journal of Educational and Behavioral Statistics最新文献_第7页

Estimating Heterogeneous Treatment Effects Within Latent Class Multilevel Models: A Bayesian Approach 在潜在类多水平模型中估计异质性治疗效果:贝叶斯方法

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-08-17 DOI: 10.3102/10769986221115446

Weicong Lyu, Jee-Seon Kim, Youmi Suk

引用次数: 0

Cognitive Diagnosis Modeling Incorporating Response Times and Fixation Counts: Providing Comprehensive Feedback and Accurate Diagnosis 结合反应时间和注视次数的认知诊断模型：提供全面反馈和准确诊断

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-28 DOI: 10.3102/10769986221111085

P. Zhan, K. Man, Stefanie A. Wind, Jonathan Malone

{"title":"Cognitive Diagnosis Modeling Incorporating Response Times and Fixation Counts: Providing Comprehensive Feedback and Accurate Diagnosis","authors":"P. Zhan, K. Man, Stefanie A. Wind, Jonathan Malone","doi":"10.3102/10769986221111085","DOIUrl":"https://doi.org/10.3102/10769986221111085","url":null,"abstract":"Respondents’ problem-solving behaviors comprise behaviors that represent complicated cognitive processes that are frequently systematically tied to one another. Biometric data, such as visual fixation counts (FCs), which are an important eye-tracking indicator, can be combined with other types of variables that reflect different aspects of problem-solving behavior to quantify variability in problem-solving behavior. To provide comprehensive feedback and accurate diagnosis when using such multimodal data, the present study proposes a multimodal joint cognitive diagnosis model that accounts for latent attributes, latent ability, processing speed, and visual engagement by simultaneously modeling response accuracy (RA), response times, and FCs. We used two simulation studies to test the feasibility of the proposed model. Findings mainly suggest that the parameters of the proposed model can be well recovered and that modeling FCs, in addition to RA and response times, could increase the comprehensiveness of feedback on problem-solving-related cognitive characteristics as well as the accuracy of knowledge structure diagnosis. An empirical example is used to demonstrate the applicability and benefits of the proposed model. We discuss the implications of our findings as they relate to research and practice.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"736 - 776"},"PeriodicalIF":2.4,"publicationDate":"2022-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47269107","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 8

Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression 使用稳健回归测试没有预定义锚项目的差异项目功能

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-18 DOI: 10.3102/10769986221109208

Weimeng Wang, Yang Liu, Hongyun Liu

{"title":"Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression","authors":"Weimeng Wang, Yang Liu, Hongyun Liu","doi":"10.3102/10769986221109208","DOIUrl":"https://doi.org/10.3102/10769986221109208","url":null,"abstract":"Differential item functioning (DIF) occurs when the probability of endorsing an item differs across groups for individuals with the same latent trait level. The presence of DIF items may jeopardize the validity of an instrument; therefore, it is crucial to identify DIF items in routine operations of educational assessment. While DIF detection procedures based on item response theory (IRT) have been widely used, a majority of IRT-based DIF tests assume predefined anchor (i.e., DIF-free) items. Not only is this assumption strong, but violations to it may also lead to erroneous inferences, for example, an inflated Type I error rate. We propose a general framework to define the effect sizes of DIF without a priori knowledge of anchor items. In particular, we quantify DIF by item-specific residuals from a regression model fitted to the true item parameters in respective groups. Moreover, the null distribution of the proposed test statistic using robust estimator can be derived analytically or approximated numerically even when there is a mix of DIF and non-DIF items, which yields asymptotically justified statistical inference. The Type I error rate and the power performance of the proposed procedure are evaluated and compared with the conventional likelihood-ratio DIF tests in a Monte Carlo experiment. Our simulation study has shown promising results in controlling Type I error rate and power of detecting DIF items. Even when there is a mix of DIF and non-DIF items, the true and false alarm rate can be well controlled when a robust regression estimator is used.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"666 - 692"},"PeriodicalIF":2.4,"publicationDate":"2022-07-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42754815","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 8

Zero and One Inflated Item Response Theory Models for Bounded Continuous Data 有界连续数据的零和一充气项目反应理论模型

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-15 DOI: 10.3102/10769986221108455

D. Molenaar, M. Curi, Jorge L. Bazán

{"title":"Zero and One Inflated Item Response Theory Models for Bounded Continuous Data","authors":"D. Molenaar, M. Curi, Jorge L. Bazán","doi":"10.3102/10769986221108455","DOIUrl":"https://doi.org/10.3102/10769986221108455","url":null,"abstract":"Bounded continuous data are encountered in many applications of item response theory, including the measurement of mood, personality, and response times and in the analyses of summed item scores. Although different item response theory models exist to analyze such bounded continuous data, most models assume the data to be in an open interval and cannot accommodate data in a closed interval. As a result, ad hoc transformations are needed to prevent scores on the bounds of the observed variables. To motivate the present study, we demonstrate in real and simulated data that this practice of fitting open interval models to closed interval data can majorly affect parameter estimates even in cases with only 5% of the responses on one of the bounds of the observed variables. To address this problem, we propose a zero and one inflated item response theory modeling framework for bounded continuous responses in the closed interval. We illustrate how four existing models for bounded responses from the literature can be accommodated in the framework. The resulting zero and one inflated item response theory models are studied in a simulation study and a real data application to investigate parameter recovery, model fit, and the consequences of fitting the incorrect distribution to the data. We find that neglecting the bounded nature of the data biases parameters and that misspecification of the exact distribution may affect the results depending on the data generating model.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"693 - 735"},"PeriodicalIF":2.4,"publicationDate":"2022-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45894141","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Forced-Choice Ranking Models for Raters’ Ranking Data 评分者排名数据的强迫选择排名模型

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-07 DOI: 10.3102/10769986221104207

Su-Pin Hung, Hung-Yu Huang

{"title":"Forced-Choice Ranking Models for Raters’ Ranking Data","authors":"Su-Pin Hung, Hung-Yu Huang","doi":"10.3102/10769986221104207","DOIUrl":"https://doi.org/10.3102/10769986221104207","url":null,"abstract":"To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees’ performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters’ leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"603 - 634"},"PeriodicalIF":2.4,"publicationDate":"2022-07-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46199623","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables 异质方差分量模型评估评分者间可靠性：考虑上下文变量的灵活方法

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-05 DOI: 10.3102/10769986221150517

Patrícia Martinková, František Bartoš, M. Brabec

{"title":"Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables","authors":"Patrícia Martinková, František Bartoš, M. Brabec","doi":"10.3102/10769986221150517","DOIUrl":"https://doi.org/10.3102/10769986221150517","url":null,"abstract":"Inter-rater reliability (IRR), which is a prerequisite of high-quality ratings and assessments, may be affected by contextual variables, such as the rater’s or ratee’s gender, major, or experience. Identification of such heterogeneity sources in IRR is important for the implementation of policies with the potential to decrease measurement error and to increase IRR by focusing on the most relevant subgroups. In this study, we propose a flexible approach for assessing IRR in cases of heterogeneity due to covariates by directly modeling differences in variance components. We use Bayes factors (BFs) to select the best performing model, and we suggest using Bayesian model averaging as an alternative approach for obtaining IRR and variance component estimates, allowing us to account for model uncertainty. We use inclusion BFs considering the whole model space to provide evidence for or against differences in variance components due to covariates. The proposed method is compared with other Bayesian and frequentist approaches in a simulation study, and we demonstrate its superiority in some situations. Finally, we provide real data examples from grant proposal peer review, demonstrating the usefulness of this method and its flexibility in the generalization of more complex designs.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"48 1","pages":"349 - 383"},"PeriodicalIF":2.4,"publicationDate":"2022-07-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46141747","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Pooling Interactions Into Error Terms in Multisite Experiments 在多站点实验中将相互作用合并为误差项

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-07-04 DOI: 10.3102/10769986221104800

Wendy Chan, L. Hedges

引用次数: 1

Improving Accuracy and Stability of Aggregate Student Growth Measures Using Empirical Best Linear Prediction 使用经验最佳线性预测提高学生成长总量测量的准确性和稳定性

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-06-27 DOI: 10.3102/10769986221101624

J. R. Lockwood, K. Castellano, D. McCaffrey

引用次数: 1

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings 速度-精度权衡？不那么快：在现实世界中，速度的边际变化与准确性之间存在不一致的关系

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-06-08 DOI: 10.3102/10769986221099906

B. Domingue, K. Kanopka, B. Stenhaug, M. Sulik, Tanesia Beverly, Matthieu J. S. Brinkhuis, Ruhan Circi, Jessica Faul, Dandan Liao, Bruce McCandliss, Jelena Obradović, Chris Piech, Tenelle Porter, Project iLEAD Consortium, J. Soland, Jon Weeks, S. Wise, Jason D Yeatman

{"title":"Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings","authors":"B. Domingue, K. Kanopka, B. Stenhaug, M. Sulik, Tanesia Beverly, Matthieu J. S. Brinkhuis, Ruhan Circi, Jessica Faul, Dandan Liao, Bruce McCandliss, Jelena Obradović, Chris Piech, Tenelle Porter, Project iLEAD Consortium, J. Soland, Jon Weeks, S. Wise, Jason D Yeatman","doi":"10.3102/10769986221099906","DOIUrl":"https://doi.org/10.3102/10769986221099906","url":null,"abstract":"The speed–accuracy trade-off (SAT) suggests that time constraints reduce response accuracy. Its relevance in observational settings—where response time (RT) may not be constrained but respondent speed may still vary—is unclear. Using 29 data sets containing data from cognitive tasks, we use a flexible method for identification of the SAT (which we test in extensive simulation studies) to probe whether the SAT holds. We find inconsistent relationships between time and accuracy; marginal increases in time use for an individual do not necessarily predict increases in accuracy. Additionally, the speed–accuracy relationship may depend on the underlying difficulty of the interaction. We also consider the analysis of items and individuals; of particular interest is the observation that respondents who exhibit more within-person variation in response speed are typically of lower ability. We further find that RT is typically a weak predictor of response accuracy. Our findings document a range of empirical phenomena that should inform future modeling of RTs collected in observational settings.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"576 - 602"},"PeriodicalIF":2.4,"publicationDate":"2022-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48011887","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

What Is Actually Equated in “Test Equating”? A Didactic Note 在“测试等价”中，什么是真正等价的?说教笔记

IF 2.4 3区心理学

Journal of Educational and Behavioral Statistics Pub Date : 2022-06-01 DOI: 10.3102/10769986211072308

Wim J. van der Linden

引用次数: 1