Wanjing Anya Ma, Adam Richie-Halford, Amy K Burkhardt, Klint Kanopka, Clementine Chou, Benjamin W Domingue, Jason D Yeatman
{"title":"ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing.","authors":"Wanjing Anya Ma, Adam Richie-Halford, Amy K Burkhardt, Klint Kanopka, Clementine Chou, Benjamin W Domingue, Jason D Yeatman","doi":"10.3758/s13428-024-02578-y","DOIUrl":"10.3758/s13428-024-02578-y","url":null,"abstract":"<p><p>The Rapid Online Assessment of Reading (ROAR) is a web-based lexical decision task that measures single-word reading abilities in children and adults without a proctor. Here we study whether item response theory (IRT) and computerized adaptive testing (CAT) can be used to create a more efficient online measure of word recognition. To construct an item bank, we first analyzed data taken from four groups of students (N = 1960) who differed in age, socioeconomic status, and language-based learning disabilities. The majority of item parameters were highly consistent across groups (r = .78-.94), and six items that functioned differently across groups were removed. Next, we implemented a JavaScript CAT algorithm and conducted a validation experiment with 485 students in grades 1-8 who were randomly assigned to complete trials of all items in the item bank in either (a) a random order or (b) a CAT order. We found that, to achieve reliability of 0.9, CAT improved test efficiency by 40%: 75 CAT items produced the same standard error of measurement as 125 items in a random order. Subsequent validation in 32 public school classrooms showed that an approximately 3-min ROAR-CAT can achieve high correlations (r = .89 for first grade, r = .73 for second grade) with alternative 5-15-min individually proctored oral reading assessments. Our findings suggest that ROAR-CAT is a promising tool for efficiently and accurately measuring single-word reading ability. Furthermore, our development process serves as a model for creating adaptive online assessments that bridge research and practice.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"56"},"PeriodicalIF":4.6,"publicationDate":"2025-01-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11732908/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142982534","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A note on using random starting values in small sample SEM.","authors":"Julie De Jonckere, Yves Rosseel","doi":"10.3758/s13428-024-02543-9","DOIUrl":"10.3758/s13428-024-02543-9","url":null,"abstract":"<p><p>Model estimation for SEM analyses in commonly used software typically involves iterative optimization procedures, which can lead to nonconvergence issues. In this paper, we propose using random starting values as an alternative to the current default strategies. By drawing from uniform distributions within data-driven lower and upper bounds (see De Jonckere et al. (2022) Structural Equation Modeling: A Multidisciplinary Journal, 29(3), 412-427), random starting values are generated for each (free) parameter in the model. Through three small simulation studies, we demonstrate that incorporating such bounded random starting values significantly reduces the nonconvergence rate, resulting in increased convergence rates ranging between 87% and 96% in the first two studies. In essence, bounded random starting values seem to offer a promising alternative to the default starting values that are currently used in most software packages.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"57"},"PeriodicalIF":4.6,"publicationDate":"2025-01-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142982530","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Robust estimation of the latent trait in graded response models.","authors":"Audrey Filonczuk, Ying Cheng","doi":"10.3758/s13428-024-02574-2","DOIUrl":"10.3758/s13428-024-02574-2","url":null,"abstract":"<p><p>Aberrant responses (e.g., careless responses, miskeyed items, etc.) often contaminate psychological assessments and surveys. Previous robust estimators for dichotomous IRT models have produced more accurate latent trait estimates with data containing response disturbances. However, for widely used Likert-type items with three or more response categories, a robust estimator for estimating latent traits does not exist. We propose a robust estimator for the graded response model (GRM) that can be applied to Likert-type items. Two weighting mechanisms for downweighting \"suspicious\" responses are considered: the Huber and the bisquare weight functions. Simulations reveal the estimator reduces bias for various test lengths, numbers of response categories, and types of response disturbances. The reduction in bias and stable standard errors suggests that the robust estimator for the GRM is effective in counteracting the harmful effects of response disturbances and providing more accurate scores on psychological assessments. The robust estimator is then applied to data from the Big Five Inventory-2 (Ober et al., 2021) to demonstrate its use. Potential applications and implications are discussed.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"55"},"PeriodicalIF":4.6,"publicationDate":"2025-01-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142982536","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A response time-based mixture item response theory model for dynamic item-response strategies.","authors":"Sijia Huang, Jinwen Luo, Minjeong Jeon","doi":"10.3758/s13428-024-02555-5","DOIUrl":"10.3758/s13428-024-02555-5","url":null,"abstract":"<p><p>Educational researchers have a long-lasting interest in the strategies examinees employ when responding to items in an assessment. Mixture item response theory (IRT) modeling is a popular class of approaches to studying examinees' item-response strategies. In the present study, we introduce a response time (RT)-based mixture IRT model for flexible modeling of examinee-and-item-specific item-response strategies. We posit that examinees may alternate between ability-based and non-ability-based strategies across different test items. Our proposed model identifies such within-examinee strategy switches without the need to predefine the non-ability-based strategies. Instead, our proposed approach allows for inferring the nature of these strategies from model parameter estimates. We illustrated the proposed approach using empirical data from PISA 2018 Science test and evaluated it through simulation studies. We concluded the article with discussions of limitations and future research directions.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"54"},"PeriodicalIF":4.6,"publicationDate":"2025-01-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943472","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Suzanne Jak, Terrence D Jorgensen, Mathilde G E Verdam, Frans J Oort, Louise Elffers
{"title":"Author Correction: Analytical power calculations for structural equation modeling: A tutorial and Shiny app.","authors":"Suzanne Jak, Terrence D Jorgensen, Mathilde G E Verdam, Frans J Oort, Louise Elffers","doi":"10.3758/s13428-024-02571-5","DOIUrl":"10.3758/s13428-024-02571-5","url":null,"abstract":"","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"53"},"PeriodicalIF":4.6,"publicationDate":"2025-01-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11717876/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Testing measurement invariance in a conditional likelihood framework by considering multiple covariates simultaneously.","authors":"Clemens Draxler, Andreas Kurz","doi":"10.3758/s13428-024-02551-9","DOIUrl":"10.3758/s13428-024-02551-9","url":null,"abstract":"<p><p>This article addresses the problem of measurement invariance in psychometrics. In particular, its focus is on the invariance assumption of item parameters in a class of models known as Rasch models. It suggests a mixed-effects or random intercept model for binary data together with a conditional likelihood approach of both estimating and testing the effects of multiple covariates simultaneously. The procedure can also be viewed as a multivariate multiple regression analysis which can be applied in longitudinal designs to investigate effects of covariates over time or different experimental conditions. This work also derives four statistical tests based on asymptotic theory and a parameter-free test suitable in small sample size scenarios. Finally, it outlines generalizations for categorical data in more than two categories. All procedures are illustrated on real-data examples from behavioral research and on a hypothetical data example related to clinical research in a longitudinal design.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"50"},"PeriodicalIF":4.6,"publicationDate":"2025-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11711259/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943479","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Quantifying sighting dominance using on-display projections of monocular and binocular views.","authors":"Giuseppe Notaro, Uri Hasson","doi":"10.3758/s13428-024-02512-2","DOIUrl":"10.3758/s13428-024-02512-2","url":null,"abstract":"<p><p>Sighting dominance is an important behavioral property which has been difficult to measure quantitatively with high precision. We developed a measurement method that is grounded in a two-camera model that satisfies these aims. Using a simple alignment task, this method quantifies sighting ocular dominance during binocular viewing, identifying each eye's relative contribution to binocular vision. The method involves placing a physical target between the viewer and a display. The viewer indicates the perceived target's projection on the display with both eyes open and with only one eye open. The relative location of the binocular projection in relation to the two monocular projections is the index of dominance. The method produces a continuous variable with robust test-retest reliability (ICC = 0.96). The unit of measurement for the computed quantity is physiologically grounded: it is proportional to the distance between the monocular projections, which we show is predictable from interpupillary distance and phoria. Comparisons with the classic 'hole in card' sighting dominance test show good agreement, but also hint at potential bias for determining right-eye dominance in the latter. Interestingly, we find that some individuals systematically demonstrate nearly balanced vision, a phenomenon previously construed as mixed dominance or noisy responses. We also present ways to quantify and mitigate sources of random noise in this measurement. Overall, this new method allows for precise estimation of sighting dominance during binocular viewing. We expect it will allow a more effective understanding of the neural basis of dominance and improved effectiveness when using sighting dominance as a covariate in more complex analyses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"52"},"PeriodicalIF":4.6,"publicationDate":"2025-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943475","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Semantic alignment: A measure to quantify the degree of semantic equivalence for English-Chinese translation equivalents based on distributional semantics.","authors":"Yufeng Liu, Shifa Chen, Yi Yang","doi":"10.3758/s13428-024-02527-9","DOIUrl":"10.3758/s13428-024-02527-9","url":null,"abstract":"<p><p>The degree of semantic equivalence of translation pairs is typically measured by asking bilinguals to rate the semantic similarity of them or comparing the number and meaning of dictionary entries. Such measures are subjective, labor-intensive, and unable to capture the fine-grained variation in the degree of semantic equivalence. Thompson et al. (in Nature Human Behaviour, 4(10), 1029-1038, 2020) propose a computational method to quantify the extent to which translation equivalents are semantically aligned by measuring the contextual use across languages. Here, we refine this method to quantify semantic alignment of English-Chinese translation equivalents using word2vec based on the proposal that the degree of similarity between the contexts associated with a word and those of its multiple translations vary continuously. We validate our measure using semantic alignment from GloVe and fastText, and data from two behavioral datasets. The consistency of semantic alignment induced across different models confirms the robustness of our method. We demonstrate that semantic alignment not only reflects human semantic similarity judgment of translation equivalents but also captures bilinguals' usage frequency of translations. We also show that our method is more cognitively plausible than Thompson et al.'s method. Furthermore, the correlations between semantic alignment and key psycholinguistic factors mirror those between human-rated semantic similarity and these variables, indicating that computed semantic alignment reflects the degree of semantic overlap of translation equivalents in the bilingual mental lexicon. We further provide the largest English-Chinese translation equivalent dataset to date, encompassing 50,088 translation pairs for 15,734 English words, their dominant Chinese translation equivalents, and their semantic alignment Rc values.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"51"},"PeriodicalIF":4.6,"publicationDate":"2025-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Noémie Auclair-Ouellet, Alexandra Lavoie, Pascale Bédard, Alexandra Barbeau-Morrison, Patrick Drouin, Pascale Tremblay
{"title":"Expansion of the SyllabO+ corpus and database: Words, lemmas, and morphology.","authors":"Noémie Auclair-Ouellet, Alexandra Lavoie, Pascale Bédard, Alexandra Barbeau-Morrison, Patrick Drouin, Pascale Tremblay","doi":"10.3758/s13428-024-02582-2","DOIUrl":"10.3758/s13428-024-02582-2","url":null,"abstract":"<p><p>Having a detailed description of the psycholinguistic properties of a language is essential for conducting well-controlled language experiments. However, there is a paucity of databases for some languages and regional varieties, including Québec French. The SyllabO+ corpus was created to provide a complete phonological and syllabic analysis of a corpus of spoken Québec French. In the present study, the corpus was expanded with 41 additional speakers, bringing the total to 225. The analysis was also expanded to include three new databases: unique words, lemmas, and morphemes (inflectional, derivational, and compounds). Next, the internal structure of unique words was analyzed to identify roots, inflectional markers, and affixes, as well as the components of compounds. Additionally, a group of 441 speakers of Québec French provided semantic transparency ratings for 3764 derived words. Results from the semantic transparency judgment study show broad inter-individual variability for words of medium transparency. No influence of sociodemographic variables was found. Transparency ratings are coherent with studies showing the greater transparency of suffixed words compared to prefixed words. Results for participants who speak French as a second language support the association between second-language proficiency and morphological processing.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"47"},"PeriodicalIF":4.6,"publicationDate":"2025-01-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943474","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Beyond Reality Image Collection (BRIC).","authors":"Noga Segal-Gordon, Yoav Bar-Anan","doi":"10.3758/s13428-024-02586-y","DOIUrl":"10.3758/s13428-024-02586-y","url":null,"abstract":"<p><p>The Beyond Reality Image Collection (BRIC) is a set of 648 photos, some painted by an artist and some generated by artificial intelligence. Unlike previous photosets, the BRIC focused on nonrealistic visuals. This collection includes abstract and non-abstract paintings and nonrealistic photographs depicting objects, scenes, animals, humans, and fantastical creatures with varying degrees of unreal elements. We collected evaluative ratings of the photos, using a convenience sample of 16,208 participants in a total of 25,321 sessions. We used multiple evaluation measures: binary positive/negative and like/dislike categorization, seven-point ratings on these attributes, both under no time pressure and under time pressure, and evaluative priming scores. The mean evaluation of the photos on the different measures was highly correlated, but some photos consistently elicited a discrepant evaluative reaction between the measures. The BRIC is a valuable resource for eliciting evaluative reactions and can contribute to research on evaluative processes and affective responses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 1","pages":"49"},"PeriodicalIF":4.6,"publicationDate":"2025-01-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11706899/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142943480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}