Behavior Research Methods最新文献

{"title":"Accounting for item calibration error in computerized adaptive testing.","authors":"Aron Fink, Christoph König, Andreas Frey","doi":"10.3758/s13428-025-02649-8","DOIUrl":"10.3758/s13428-025-02649-8","url":null,"abstract":"<p><p>In computerized adaptive testing (CAT), item parameter estimates derived from calibration studies are considered to be known and are used as fixed values for adaptive item selection and ability estimation. This is not completely accurate because these item parameter estimates contain a certain degree of error. If this error is random, the typical CAT procedure leads to standard errors of the final ability estimates that are too small. If the calibration error is large, it has been shown that the accuracy of the ability estimates is negatively affected due to the capitalization on chance problem, especially for extreme ability levels. In order to find a solution for this fundamental problem of CAT, we conducted a Monte Carlo simulation study to examine three approaches that can be used to consider the uncertainty of item parameter estimates in CAT. The first two approaches used a measurement error modeling approach in which item parameters were treated as covariates that contained errors. The third approach was fully Bayesian. Each of the approaches was compared with regard to the quality of the resulting ability estimates. The results indicate that each of the three approaches is capable of reducing bias and the mean squared error (MSE) of the ability estimates, especially for high item calibration errors. The Bayesian approach clearly outperformed the other approaches. We recommend the Bayesian approach, especially for application areas in which the recruitment of a large calibration sample is infeasible.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 5","pages":"126"},"PeriodicalIF":4.6,"publicationDate":"2025-03-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11947018/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143717853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Sample size matters when estimating test-retest reliability of behaviour.","authors":"Brendan Williams, Lily FitzGibbon, Daniel Brady, Anastasia Christakou","doi":"10.3758/s13428-025-02599-1","DOIUrl":"10.3758/s13428-025-02599-1","url":null,"abstract":"<p><p>Intraclass correlation coefficients (ICCs) are a commonly used metric in test-retest reliability research to assess a measure's ability to quantify systematic between-subject differences. However, estimates of between-subject differences are also influenced by factors including within-subject variability, random errors, and measurement bias. Here, we use data collected from a large online sample (N = 150) to (1) quantify test-retest reliability of behavioural and computational measures of reversal learning using ICCs, and (2) use our dataset as the basis for a simulation study investigating the effects of sample size on variance component estimation and the association between estimates of variance components and ICC measures. In line with previously published work, we find reliable behavioural and computational measures of reversal learning, a commonly used assay of behavioural flexibility. Reliable estimates of between-subject, within-subject (across-session), and error variance components for behavioural and computational measures (with ± .05 precision and 80% confidence) required sample sizes ranging from 10 to over 300 (behavioural median N: between-subject = 167, within-subject = 34, error = 103; computational median N: between-subject = 68, within-subject = 20, error = 45). These sample sizes exceed those often used in reliability studies, suggesting that sample sizes larger than are commonly used for reliability studies (circa 30) are required to robustly estimate reliability of task performance measures. Additionally, we found that ICC estimates showed highly positive and highly negative correlations with between-subject and error variance components, respectively, as might be expected, which remained relatively stable across sample sizes. However, ICC estimates were weakly or not correlated with within-subject variance, providing evidence for the importance of variance decomposition for reliability studies.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"123"},"PeriodicalIF":4.6,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11928395/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143676714","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ESM-Q: A consensus-based quality assessment tool for experience sampling method items.","authors":"Gudrun Eisele, Anu Hiekkaranta, Yoram K Kunkels, Marije Aan Het Rot, Wouter van Ballegooijen, Sara Laureen Bartels, Jojanneke A Bastiaansen, Patrick N Beymer, Lauren M Bylsma, Ryan W Carpenter, William D Ellison, Aaron J Fisher, Thomas Forkmann, Madelyn R Frumkin, Daniel Fulford, Kristin Naragon-Gainey, Talya Greene, Vera E Heininga, Andrew Jones, Elise K Kalokerinos, Peter Kuppens, Kathryn L Modecki, Fabiola Müller, Andreas B Neubauer, Vanessa Panaite, Maude Schneider, Jessie Sun, Stephen J Wilson, Caroline Zygar-Hoffmann, Inez Myin-Germeys, Olivia J Kirtley","doi":"10.3758/s13428-025-02626-1","DOIUrl":"10.3758/s13428-025-02626-1","url":null,"abstract":"<p><p>The experience sampling method (ESM) is increasingly used by researchers from various disciplines to answer novel questions about individuals' daily lives. Measurement best practices have long been overlooked in ESM research, and recent reviews show that item quality is often not reported in ESM studies. The absence of information about item quality may be partly explained by the lack of consensus on how ESM item quality should be evaluated. As part of the ESM Item Repository project (esmitemrepository.com)-an international open science initiative that collects ESM items in an open item bank and evaluates their quality-we brought together 42 international ESM experts to develop an ESM item quality assessment tool. In four Delphi phases, experts suggested 57 item quality criteria, rated the criteria, provided arguments for and against the criteria, and rated the criteria again, considering reflections from other experts. The result of the Delphi process is ESM-Q: a quality assessment tool consisting of 10 core criteria, as well as an additional 15 supplementary criteria, to be used depending on the type of items being rated and the availability of supplementary information. The criteria cover topics ranging from construct validity to the optimal wording of items. ESM-Q can aid ESM researchers in selecting existing ESM items, developing new high-quality ESM items, and evaluating the quality of ESM items in systematic reviews. Expert reflections also highlight open research questions surrounding ESM item design that form a research agenda for ESM measurement.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"124"},"PeriodicalIF":4.6,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143676668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What you see is not what you get: Observed scale score comparisons misestimate true group differences.","authors":"Bjarne Schmalbach, Ileana Schmalbach, Jochen Hardt","doi":"10.3758/s13428-025-02639-w","DOIUrl":"10.3758/s13428-025-02639-w","url":null,"abstract":"<p><p>Social sciences of all kinds are interested in latent variables, their measurement, and how they differ between groups. The present study argues the importance of analyzing mean differences between groups using the latent variable approach. Using an open-access repository of widely applied personality questionnaires (N = 999,033), we evaluate the extent to which the commonly used observed sum score is susceptible to measurement error. Our findings show that Cohen's d values based on the observed variance significantly misestimate the true group difference (based on just the factor score variance) in 33 of the 70 studied cases, and by an average of 25.0% (or 0.048 standard deviations). There was no meaningful relationship between the effect size discrepancy and scale reliability as measured by McDonald's ω. We discuss the implications of these results and outline concrete steps that applied researchers can take to improve their analyses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"122"},"PeriodicalIF":4.6,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11923020/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143662330","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Appropriate modeling of endogeneity in cross-lagged models: Efficacy of auxiliary and model-implied instrumental variables.","authors":"Junyan Fang, Zhonglin Wen, Kit-Tai Hau, Xitong Huang","doi":"10.3758/s13428-025-02631-4","DOIUrl":"10.3758/s13428-025-02631-4","url":null,"abstract":"<p><p>Endogeneity is a critical concern in research methodologies, yet it has been insufficiently addressed in longitudinal cross-lagged models, leading to potentially biased outcomes. This study scrutinized the endogeneity inherent in the cross-lagged panel model (CLPM), a prevalent and representative framework in longitudinal studies. We evaluated the efficacy of the instrumental variables (IV) methods, specifically focusing on both the auxiliary IVs (AIVs) and the model-implied IVs (MIIVs), in mitigating endogeneity issues. Simulation results indicated that endogeneity induced bias in CLPM, notably overestimating cross-lagged effects and thereby amplifying the apparent causal relationships. AIV-CLPM showed a smaller, yet still unacceptably high bias, along with low robustness and elevated type I error rates. In contrast, the MIIV-CLPM produced more accurate estimates with fewer type I errors, and, given sufficient observations, it achieved moderate statistical power. An extended simulation incorporating the random-intercept CLPM supported these findings, highlighting the generalizability of this approach. Furthermore, an empirical illustration demonstrated the practicality and feasibility of the MIIV-CLPM. Overall, MIIV is proven to be a superior modeling option within cross-lagged frameworks, effectively mitigating biases caused by endogeneity.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"121"},"PeriodicalIF":4.6,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143662318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Ready to ROC? A tutorial on simulation-based power analyses for null hypothesis significance, minimum-effect, and equivalence testing for ROC curve analyses.","authors":"Paul Riesthuis, Henry Otgaar, Charlotte Bücken","doi":"10.3758/s13428-025-02646-x","DOIUrl":"10.3758/s13428-025-02646-x","url":null,"abstract":"<p><p>The receiver operating characteristic (ROC) curve and its corresponding (partial) area under the curve (AUC) are frequently used statistical tools in psychological research to assess the discriminability of a test, method, intervention, or procedure. In this paper, we provide a tutorial on conducting simulation-based power analyses for ROC curve and (p)AUC analyses in R. We also created a Shiny app and the R package \"ROCpower\" to perform such power analyses. In our tutorial, we highlight the importance of setting the smallest effect size of interest (SESOI) for which researchers want to conduct their power analysis. The SESOI is the smallest effect that is practically or theoretically relevant for a specific field of research or study. We provide how such a SESOI can be established and how it changes hypotheses from simply establishing whether there is a statistically significant effect (i.e., null-hypothesis significance testing) to whether the effects are practically or theoretically important (i.e., minimum-effect testing) or whether the effect is too small to care about (i.e., equivalence testing). We show how power analyses for these different hypothesis tests can be conducted via a confidence interval-focused approach. This confidence interval-focused, simulation-based power analysis can be adapted to different research designs and questions and improves the reproducibility of power analyses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"120"},"PeriodicalIF":4.6,"publicationDate":"2025-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11920309/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143656171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance of location-scale models in meta-analysis: A simulation study.","authors":"Desirée Blázquez-Rincón, José Antonio López-López, Wolfgang Viechtbauer","doi":"10.3758/s13428-025-02622-5","DOIUrl":"10.3758/s13428-025-02622-5","url":null,"abstract":"<p><p>Location-scale models in the field of meta-analysis allow researchers to simultaneously study the influence of moderator variables on the mean (location) and variance (scale) of the distribution of true effects. However, the increased complexity of such models can make model fitting challenging. Moreover, the statistical properties of the estimation and inference methods for such models have not been systematically examined in the meta-analytic context. We therefore conducted a Monte Carlo simulation study to compare different estimation methods (maximum or restricted maximum likelihood estimation), significance tests (Wald-type, permutation, and likelihood-ratio tests), and methods for constructing confidence intervals (Wald-type and profile-likelihood intervals) for the scale coefficients of such models. When restricted maximum likelihood estimation was used, slightly closer to nominal rejection rates and narrower confidence intervals were obtained. The permutation test yielded type I error rates closest to the nominal level, whereas the likelihood-ratio test obtained the highest statistical power. In most scenarios, profile-likelihood intervals showed lower coverage probabilities than the Wald-type method but closer to the nominal 95% level. Finally, slightly higher rejection rates and coverage probabilities were obtained when a dichotomous moderator was examined rather than a continuous one. Despite the need to use some constraints on the parameter space for the scale coefficients and the possibility of non-convergence of some procedures that may affect the fitting of the specified models, location-scale models proved to be a valid and useful tool for modeling the heterogeneity parameter in meta-analysis.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"118"},"PeriodicalIF":4.6,"publicationDate":"2025-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11914364/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143647006","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Samply Stream API: The AI-enhanced method for real-time event data streaming.","authors":"Yury Shevchenko, Ulf-Dietrich Reips","doi":"10.3758/s13428-025-02634-1","DOIUrl":"10.3758/s13428-025-02634-1","url":null,"abstract":"<p><p>This manuscript introduces a novel method for conducting behavioral and social research by streaming real-time information to participants and manipulating content for experimental purposes via AI. We present an extension of the Samply software, which facilitates the integration of event-related data with mobile surveys and experiments. To assess the feasibility of this method, we conducted an experiment where news headlines were modified by a Chat-GPT algorithm and streamed to participants via the Samply Stream API and mobile push notifications. Feedback from participants indicated that most did not experience technical problems. There was no significant difference in readability across original, paraphrased, and misinformation-injected news conditions, with only 1.2% of all news items reported as unreadable. Participants reported significantly less familiarity with misinformation-injected news (84% unfamiliarity) compared to original and paraphrased news (73% unfamiliarity), suggesting successful manipulation of information without compromising readability. Dropout and non-response rates were comparable to those in other experience sampling studies. The streaming method offers significant potential for various applications, including public opinion research, healthcare, marketing, and environmental monitoring. By enabling the real-time collection of contextually relevant data, this method has the potential to enhance the external validity of behavioral research and provides a powerful tool for studying human behavior in naturalistic settings.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"119"},"PeriodicalIF":4.6,"publicationDate":"2025-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11914333/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143647012","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Revised network loadings.","authors":"Alexander P Christensen, Hudson Golino, Francisco J Abad, Luis Eduardo Garrido","doi":"10.3758/s13428-025-02640-3","DOIUrl":"10.3758/s13428-025-02640-3","url":null,"abstract":"<p><p>Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which aim to provide network models with a metric similar to factor loadings to assess measurement quality when the data are generated from a factor model. Our study revisits and refines the original network loadings to account for properties of (regularized) partial correlation networks, such as the reduction of partial correlation size as the number of variables increase, that were not considered previously. Using a simulation study, the revised network loadings demonstrated greater congruence with the simulated factor loadings across conditions relative to the original formulation. The simulation also evaluated how well correlations between factors can be captured by scores estimated with network loadings. The results show that not only can these network scores adequately estimate the simulated correlations between factors, they can do so without the need for rotation, a standard requirement for factor loadings. The consequence is that researchers do not need to choose a rotation with the revised network loadings, reducing the analytic degrees of freedom and eliminating this common source of variability in factor analysis. We discuss the interpretation of network loadings when data are believed to be generated from a network model and how they may fit into a network theory of measurement.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"114"},"PeriodicalIF":4.6,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11909041/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143633309","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Unraveling other-race face perception with GAN-based image reconstruction.","authors":"Moaz Shoura, Dirk B Walther, Adrian Nestor","doi":"10.3758/s13428-025-02636-z","DOIUrl":"10.3758/s13428-025-02636-z","url":null,"abstract":"<p><p>The other-race effect (ORE) is the disadvantage of recognizing faces of another race than one's own. While its prevalence is behaviorally well documented, the representational basis of ORE remains unclear. This study employs StyleGAN2, a deep learning technique for generating photorealistic images to uncover face representations and to investigate ORE's representational basis. To this end, we collected pairwise visual similarity ratings with same- and other-race faces across East Asian and White participants exhibiting robust levels of ORE. Leveraging the significant overlap in representational similarity between the GAN's latent space and perceptual representations in human participants, we designed an image reconstruction approach aiming to reveal internal face representations from behavioral similarity data. This methodology yielded hyper-realistic depictions of face percepts, with reconstruction accuracy well above chance, as well as an accuracy advantage for same-race over other-race reconstructions, which mirrored ORE in both populations. Further, a comparison of reconstructions across participant race revealed a novel age bias, with other-race face reconstructions appearing younger than their same-race counterpart. Thus, our work proposes a new approach to exploiting the utility of GANs in image reconstruction and provides new avenues in the study of ORE.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"115"},"PeriodicalIF":4.6,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143633310","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}