Educational and Psychological Measurement最新文献

Investigating the Ordering Structure of Clustered Items Using Nonparametric Item Response Theory 利用非参数项目反应理论研究聚类项目的排序结构

IF 2.7 3区心理学

Educational and Psychological Measurement Pub Date : 2024-09-06 DOI: 10.1177/00131644241274122

Letty Koopman, Johan Braeken

{"title":"Investigating the Ordering Structure of Clustered Items Using Nonparametric Item Response Theory","authors":"Letty Koopman, Johan Braeken","doi":"10.1177/00131644241274122","DOIUrl":"https://doi.org/10.1177/00131644241274122","url":null,"abstract":"Educational and psychological tests with an ordered item structure enable efficient test administration procedures and allow for intuitive score interpretation and monitoring. The effectiveness of the measurement instrument relies to a large extent on the validated strength of its ordering structure. We define three increasingly strict types of ordering for the ordering structure of a measurement instrument with clustered items: a weak and a strong invariant cluster ordering and a clustered invariant item ordering. Following a nonparametric item response theory (IRT) approach, we proposed a procedure to evaluate the ordering structure of a clustered item set along this three-fold continuum of order invariance. The basis of the procedure is (a) the local assessment of pairwise conditional expectations at both cluster and item level and (b) the global assessment of the number of Guttman errors through new generalizations of the H-coefficient for this item-cluster context. The procedure, readily implemented in R, is illustrated and applied to an empirical example. Suggestions for test practice, further methodological developments, and future research are discussed.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":null,"pages":null},"PeriodicalIF":2.7,"publicationDate":"2024-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142178007","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Added Value of Subscores for Tests With Polytomous Items 多项式项目测试的子分数附加值

IF 2.7 3区心理学

Educational and Psychological Measurement Pub Date : 2024-08-07 DOI: 10.1177/00131644241268128

Kylie Gorney, Sandip Sinharay

引用次数: 0

A Relative Normed Effect-Size Difference Index for Determining the Number of Common Factors in Exploratory Solutions. 用于确定探索性解决方案中常见因素数量的相对规范化效应大小差异指数

IF 2.1 3区心理学

Educational and Psychological Measurement Pub Date : 2024-08-01 Epub Date: 2023-09-07 DOI: 10.1177/00131644231196482

Pere J Ferrando, David Navarro-González, Urbano Lorenzo-Seva

{"title":"A Relative Normed Effect-Size Difference Index for Determining the Number of Common Factors in Exploratory Solutions.","authors":"Pere J Ferrando, David Navarro-González, Urbano Lorenzo-Seva","doi":"10.1177/00131644231196482","DOIUrl":"10.1177/00131644231196482","url":null,"abstract":"Descriptive fit indices that do not require a formal statistical basis and do not specifically depend on a given estimation criterion are useful as auxiliary devices for judging the appropriateness of unrestricted or exploratory factor analytical (UFA) solutions, when the problem is to decide the most appropriate number of common factors. While overall indices of this type are well known in UFA applications, especially those intended for item analysis, difference indices are much more scarce. Recently, Raykov and collaborators proposed a family of effect-size-type descriptive difference indices that are promising for UFA applications. As a starting point, we considered the simplest measure of this family, which (a) can be viewed as absolute and (b) from which only tentative cutoffs and reference values have been provided so far. In this situation, this article has three aims. The first is to propose a relative version of Raykov's effect-size measure, intended to be used as a complement of the original measure, in which the increase in explained common variance is related to the overall prior estimated amount of common factor variance. The second is to establish reference values for both indices in item-analysis scenarios using simulation. And the third aim (instrumental) is to implement the proposal in both R language and a well-known non-commercial factor analysis program. The functioning and usefulness of the proposal is illustrated using an existing empirical dataset.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11268389/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46343630","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An Illustration of an IRTree Model for Disengagement. 一种用于脱离的IRTree模型说明

IF 2.1 3区心理学

Educational and Psychological Measurement Pub Date : 2024-08-01 Epub Date: 2023-07-26 DOI: 10.1177/00131644231185533

Brian C Leventhal, Dena Pastor

{"title":"An Illustration of an IRTree Model for Disengagement.","authors":"Brian C Leventhal, Dena Pastor","doi":"10.1177/00131644231185533","DOIUrl":"10.1177/00131644231185533","url":null,"abstract":"Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11268386/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45249200","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

An Ensemble Learning Approach Based on TabNet and Machine Learning Models for Cheating Detection in Educational Tests. 基于表网和机器学习模型的教育考试作弊检测集成学习方法

IF 2.1 3区心理学

Educational and Psychological Measurement Pub Date : 2024-08-01 Epub Date: 2023-08-21 DOI: 10.1177/00131644231191298

Yang Zhen, Xiaoyan Zhu

{"title":"An Ensemble Learning Approach Based on TabNet and Machine Learning Models for Cheating Detection in Educational Tests.","authors":"Yang Zhen, Xiaoyan Zhu","doi":"10.1177/00131644231191298","DOIUrl":"10.1177/00131644231191298","url":null,"abstract":"The pervasive issue of cheating in educational tests has emerged as a paramount concern within the realm of education, prompting scholars to explore diverse methodologies for identifying potential transgressors. While machine learning models have been extensively investigated for this purpose, the untapped potential of TabNet, an intricate deep neural network model, remains uncharted territory. Within this study, a comprehensive evaluation and comparison of 12 base models (naive Bayes, linear discriminant analysis, Gaussian process, support vector machine, decision tree, random forest, Extreme Gradient Boosting (XGBoost), AdaBoost, logistic regression, k-nearest neighbors, multilayer perceptron, and TabNet) was undertaken to scrutinize their predictive capabilities. The area under the receiver operating characteristic curve (AUC) was employed as the performance metric for evaluation. Impressively, the findings underscored the supremacy of TabNet (AUC = 0.85) over its counterparts, signifying the profound aptitude of deep neural network models in tackling tabular tasks, such as the detection of academic dishonesty. Encouraged by these outcomes, we proceeded to synergistically amalgamate the two most efficacious models, TabNet (AUC = 0.85) and AdaBoost (AUC = 0.81), resulting in the creation of an ensemble model christened TabNet-AdaBoost (AUC = 0.92). The emergence of this novel hybrid approach exhibited considerable potential in research endeavors within this domain. Importantly, our investigation has unveiled fresh insights into the utilization of deep neural network models for the purpose of identifying cheating in educational tests.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11268385/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41801931","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating Imputation-Based Fit Statistics in Structural Equation Modeling With Ordinal Data: The MI2S Approach 在使用序数数据的结构方程建模中评估基于估算的拟合统计量：MI2S 方法

IF 2.1 3区心理学

Educational and Psychological Measurement Pub Date : 2024-07-27 DOI: 10.1177/00131644241261271

Suppanut Sriutaisuk, Yu Liu, Seungwon Chung, Hanjoe Kim, Fei Gu

引用次数: 0

Can One Pool Over Site in a Multi-Site Study With Categorical Item Measuring Instruments?: A Multiple Testing Procedure 在使用分类项目测量工具进行的多站点研究中，能否在多个站点之间建立一个集合？多重测试程序

IF 2.1 3区心理学

Educational and Psychological Measurement Pub Date : 2024-07-27 DOI: 10.1177/00131644241267010

T. Raykov, Khaled Alkherainej

引用次数: 0

Evaluating The Predictive Reliability of Neural Networks in Psychological Research With Random Datasets 利用随机数据集评估神经网络在心理学研究中的预测可靠性

IF 2.7 3区心理学

Educational and Psychological Measurement Pub Date : 2024-07-25 DOI: 10.1177/00131644241262964

Yongtian Cheng, K. V. Petrides

{"title":"Evaluating The Predictive Reliability of Neural Networks in Psychological Research With Random Datasets","authors":"Yongtian Cheng, K. V. Petrides","doi":"10.1177/00131644241262964","DOIUrl":"https://doi.org/10.1177/00131644241262964","url":null,"abstract":"Psychologists are emphasizing the importance of predictive conclusions. Machine learning methods, such as supervised neural networks, have been used in psychological studies as they naturally fit prediction tasks. However, we are concerned about whether neural networks fitted with random datasets (i.e., datasets where there is no relationship between ordinal independent variables and continuous or binary-dependent variables) can provide an acceptable level of predictive performance from a psychologist’s perspective. Through a Monte Carlo simulation study, we found that this kind of erroneous conclusion is not likely to be drawn as long as the sample size is larger than 50 with continuous-dependent variables. However, when the dependent variable is binary, the minimum sample size is 500 when the criteria are balanced accuracy ≥ .6 or balanced accuracy ≥ .65, and the minimum sample size is 200 when the criterion is balanced accuracy ≥ .7 for a decision error less than .05. In the case where area under the curve (AUC) is used as a metric, a sample size of 100, 200, and 500 is necessary when the minimum acceptable performance level is set at AUC ≥ .7, AUC ≥ .65, and AUC ≥ .6, respectively. The results found by this study can be used for sample size planning for psychologists who wish to apply neural networks for a qualitatively reliable conclusion. Further directions and limitations of the study are also discussed.","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":null,"pages":null},"PeriodicalIF":2.7,"publicationDate":"2024-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141772234","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Studying Factorial Invariance With Nominal Items: A Note on a Latent Variable Modeling Procedure 用名义项目研究因子不变量：关于潜在变量建模程序的说明

IF 2.7 3区心理学

Educational and Psychological Measurement Pub Date : 2024-06-24 DOI: 10.1177/00131644241256626

Tenko Raykov

引用次数: 0

A Note on Evaluation of Polytomous Item Locations With the Rating Scale Model and Testing Its Fit 用评分量表模型评估多项式项目位置并测试其拟合度的说明

IF 2.7 3区心理学

Educational and Psychological Measurement Pub Date : 2024-06-24 DOI: 10.1177/00131644241259026

Tenko Raykov, Martin Pusic

引用次数: 0