Quantitative and computational methods in behavioral sciences最新文献

筛选

英文中文

Independent validation as a validation method for classification 作为分类验证方法的独立验证

Quantitative and computational methods in behavioral sciences Pub Date : 2023-12-22 DOI: 10.5964/qcmb.12069

Tina Braun, Hannes Eckert, Timo von Oertzen

{"title":"Independent validation as a validation method for classification","authors":"Tina Braun, Hannes Eckert, Timo von Oertzen","doi":"10.5964/qcmb.12069","DOIUrl":"https://doi.org/10.5964/qcmb.12069","url":null,"abstract":"The use of classifiers provides an alternative to conventional statistical methods. This involves using the accuracy with which data is correctly assigned to a given group by the classifier to apply tests to compare the performance of classifiers. The conventional validation methods for determining the accuracy of classifiers have the disadvantage that the distribution of correct classifications does not follow any known distribution, and therefore, the application of statistical tests is problematic. Independent validation circumvents this problem and allows the use of binomial tests to assess the performance of classifiers. However, independent validation accuracy is subject to bias for small training datasets. The present study shows that a hyperbolic function can be used to estimate the loss in classifier accuracy for independent validation. This function is used to develop three new methods to estimate the classifier accuracy for small training sets more precisely. These methods are compared to two existing methods in a simulation study. The results indicate overall small errors in the estimation of classifier accuracy and indicate that independent validation can be used with small samples. A least square estimation approach seems best suited to estimate the classifier accuracy.","PeriodicalId":500886,"journal":{"name":"Quantitative and computational methods in behavioral sciences","volume":"36 24","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138946720","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Estimating item parameters in multistage designs with the tmt package in R 在R语言中使用tmt包估算多级设计中的项目参数

Quantitative and computational methods in behavioral sciences Pub Date : 2023-11-06 DOI: 10.5964/qcmb.10087

Jan Steinfeld, Alexander Robitzsch

{"title":"Estimating item parameters in multistage designs with the tmt package in R","authors":"Jan Steinfeld, Alexander Robitzsch","doi":"10.5964/qcmb.10087","DOIUrl":"https://doi.org/10.5964/qcmb.10087","url":null,"abstract":"<p xmlns=\"http://www.ncbi.nlm.nih.gov/JATS1\">Various likelihood-based methods are available for the parameter estimation of item response theory models (IRT), leading to comparable parameter estimates. Considering multistage testing (MST) designs, Glas (1988; https://doi.org/10.2307/1164950) stated that the conditional maximum likelihood (CML) method in its original formulation leads to severely biased parameter estimates. A modified CML estimation method for MST designs proposed by Zwitser and Maris (2015; https://doi.org/10.1007/s11336-013-9369-6) finally provides asymptotically unbiased item parameter estimates. Steinfeld and Robitzsch (2021b; https://doi.org/10.31234/osf.io/ew27f) complemented this method to MST designs with probabilistic routing strategies. For both proposed modifications additional software solutions are required since design-specific information must be incorporated into the estimation process. An R package that has implemented both modifications is \"tmt\". In this article, first, the proposed solutions of the CML estimation in MST designs are illustrated, followed by the main part, the demonstration of the CML item parameter estimation with the R package \"tmt\". The demonstration includes the process of model specification, data simulation, and item parameter estimation, considering two different routing types of deterministic and probabilistic MST designs.","PeriodicalId":500886,"journal":{"name":"Quantitative and computational methods in behavioral sciences","volume":"19 16","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135589501","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0