Behavior Research Methods最新文献

{"title":"Sample size matters when estimating test-retest reliability of behaviour.","authors":"Brendan Williams, Lily FitzGibbon, Daniel Brady, Anastasia Christakou","doi":"10.3758/s13428-025-02599-1","DOIUrl":"10.3758/s13428-025-02599-1","url":null,"abstract":"<p><p>Intraclass correlation coefficients (ICCs) are a commonly used metric in test-retest reliability research to assess a measure's ability to quantify systematic between-subject differences. However, estimates of between-subject differences are also influenced by factors including within-subject variability, random errors, and measurement bias. Here, we use data collected from a large online sample (N = 150) to (1) quantify test-retest reliability of behavioural and computational measures of reversal learning using ICCs, and (2) use our dataset as the basis for a simulation study investigating the effects of sample size on variance component estimation and the association between estimates of variance components and ICC measures. In line with previously published work, we find reliable behavioural and computational measures of reversal learning, a commonly used assay of behavioural flexibility. Reliable estimates of between-subject, within-subject (across-session), and error variance components for behavioural and computational measures (with ± .05 precision and 80% confidence) required sample sizes ranging from 10 to over 300 (behavioural median N: between-subject = 167, within-subject = 34, error = 103; computational median N: between-subject = 68, within-subject = 20, error = 45). These sample sizes exceed those often used in reliability studies, suggesting that sample sizes larger than are commonly used for reliability studies (circa 30) are required to robustly estimate reliability of task performance measures. Additionally, we found that ICC estimates showed highly positive and highly negative correlations with between-subject and error variance components, respectively, as might be expected, which remained relatively stable across sample sizes. However, ICC estimates were weakly or not correlated with within-subject variance, providing evidence for the importance of variance decomposition for reliability studies.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"123"},"PeriodicalIF":4.6,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11928395/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143676714","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ESM-Q: A consensus-based quality assessment tool for experience sampling method items.","authors":"Gudrun Eisele, Anu Hiekkaranta, Yoram K Kunkels, Marije Aan Het Rot, Wouter van Ballegooijen, Sara Laureen Bartels, Jojanneke A Bastiaansen, Patrick N Beymer, Lauren M Bylsma, Ryan W Carpenter, William D Ellison, Aaron J Fisher, Thomas Forkmann, Madelyn R Frumkin, Daniel Fulford, Kristin Naragon-Gainey, Talya Greene, Vera E Heininga, Andrew Jones, Elise K Kalokerinos, Peter Kuppens, Kathryn L Modecki, Fabiola Müller, Andreas B Neubauer, Vanessa Panaite, Maude Schneider, Jessie Sun, Stephen J Wilson, Caroline Zygar-Hoffmann, Inez Myin-Germeys, Olivia J Kirtley","doi":"10.3758/s13428-025-02626-1","DOIUrl":"https://doi.org/10.3758/s13428-025-02626-1","url":null,"abstract":"<p><p>The experience sampling method (ESM) is increasingly used by researchers from various disciplines to answer novel questions about individuals' daily lives. Measurement best practices have long been overlooked in ESM research, and recent reviews show that item quality is often not reported in ESM studies. The absence of information about item quality may be partly explained by the lack of consensus on how ESM item quality should be evaluated. As part of the ESM Item Repository project (esmitemrepository.com)-an international open science initiative that collects ESM items in an open item bank and evaluates their quality-we brought together 42 international ESM experts to develop an ESM item quality assessment tool. In four Delphi phases, experts suggested 57 item quality criteria, rated the criteria, provided arguments for and against the criteria, and rated the criteria again, considering reflections from other experts. The result of the Delphi process is ESM-Q: a quality assessment tool consisting of 10 core criteria, as well as an additional 15 supplementary criteria, to be used depending on the type of items being rated and the availability of supplementary information. The criteria cover topics ranging from construct validity to the optimal wording of items. ESM-Q can aid ESM researchers in selecting existing ESM items, developing new high-quality ESM items, and evaluating the quality of ESM items in systematic reviews. Expert reflections also highlight open research questions surrounding ESM item design that form a research agenda for ESM measurement.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"124"},"PeriodicalIF":4.6,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143676668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What you see is not what you get: Observed scale score comparisons misestimate true group differences.","authors":"Bjarne Schmalbach, Ileana Schmalbach, Jochen Hardt","doi":"10.3758/s13428-025-02639-w","DOIUrl":"10.3758/s13428-025-02639-w","url":null,"abstract":"<p><p>Social sciences of all kinds are interested in latent variables, their measurement, and how they differ between groups. The present study argues the importance of analyzing mean differences between groups using the latent variable approach. Using an open-access repository of widely applied personality questionnaires (N = 999,033), we evaluate the extent to which the commonly used observed sum score is susceptible to measurement error. Our findings show that Cohen's d values based on the observed variance significantly misestimate the true group difference (based on just the factor score variance) in 33 of the 70 studied cases, and by an average of 25.0% (or 0.048 standard deviations). There was no meaningful relationship between the effect size discrepancy and scale reliability as measured by McDonald's ω. We discuss the implications of these results and outline concrete steps that applied researchers can take to improve their analyses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"122"},"PeriodicalIF":4.6,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11923020/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143662330","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Appropriate modeling of endogeneity in cross-lagged models: Efficacy of auxiliary and model-implied instrumental variables.","authors":"Junyan Fang, Zhonglin Wen, Kit-Tai Hau, Xitong Huang","doi":"10.3758/s13428-025-02631-4","DOIUrl":"https://doi.org/10.3758/s13428-025-02631-4","url":null,"abstract":"<p><p>Endogeneity is a critical concern in research methodologies, yet it has been insufficiently addressed in longitudinal cross-lagged models, leading to potentially biased outcomes. This study scrutinized the endogeneity inherent in the cross-lagged panel model (CLPM), a prevalent and representative framework in longitudinal studies. We evaluated the efficacy of the instrumental variables (IV) methods, specifically focusing on both the auxiliary IVs (AIVs) and the model-implied IVs (MIIVs), in mitigating endogeneity issues. Simulation results indicated that endogeneity induced bias in CLPM, notably overestimating cross-lagged effects and thereby amplifying the apparent causal relationships. AIV-CLPM showed a smaller, yet still unacceptably high bias, along with low robustness and elevated type I error rates. In contrast, the MIIV-CLPM produced more accurate estimates with fewer type I errors, and, given sufficient observations, it achieved moderate statistical power. An extended simulation incorporating the random-intercept CLPM supported these findings, highlighting the generalizability of this approach. Furthermore, an empirical illustration demonstrated the practicality and feasibility of the MIIV-CLPM. Overall, MIIV is proven to be a superior modeling option within cross-lagged frameworks, effectively mitigating biases caused by endogeneity.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"121"},"PeriodicalIF":4.6,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143662318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Ready to ROC? A tutorial on simulation-based power analyses for null hypothesis significance, minimum-effect, and equivalence testing for ROC curve analyses.","authors":"Paul Riesthuis, Henry Otgaar, Charlotte Bücken","doi":"10.3758/s13428-025-02646-x","DOIUrl":"10.3758/s13428-025-02646-x","url":null,"abstract":"<p><p>The receiver operating characteristic (ROC) curve and its corresponding (partial) area under the curve (AUC) are frequently used statistical tools in psychological research to assess the discriminability of a test, method, intervention, or procedure. In this paper, we provide a tutorial on conducting simulation-based power analyses for ROC curve and (p)AUC analyses in R. We also created a Shiny app and the R package \"ROCpower\" to perform such power analyses. In our tutorial, we highlight the importance of setting the smallest effect size of interest (SESOI) for which researchers want to conduct their power analysis. The SESOI is the smallest effect that is practically or theoretically relevant for a specific field of research or study. We provide how such a SESOI can be established and how it changes hypotheses from simply establishing whether there is a statistically significant effect (i.e., null-hypothesis significance testing) to whether the effects are practically or theoretically important (i.e., minimum-effect testing) or whether the effect is too small to care about (i.e., equivalence testing). We show how power analyses for these different hypothesis tests can be conducted via a confidence interval-focused approach. This confidence interval-focused, simulation-based power analysis can be adapted to different research designs and questions and improves the reproducibility of power analyses.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"120"},"PeriodicalIF":4.6,"publicationDate":"2025-03-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11920309/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143656171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance of location-scale models in meta-analysis: A simulation study.","authors":"Desirée Blázquez-Rincón, José Antonio López-López, Wolfgang Viechtbauer","doi":"10.3758/s13428-025-02622-5","DOIUrl":"10.3758/s13428-025-02622-5","url":null,"abstract":"<p><p>Location-scale models in the field of meta-analysis allow researchers to simultaneously study the influence of moderator variables on the mean (location) and variance (scale) of the distribution of true effects. However, the increased complexity of such models can make model fitting challenging. Moreover, the statistical properties of the estimation and inference methods for such models have not been systematically examined in the meta-analytic context. We therefore conducted a Monte Carlo simulation study to compare different estimation methods (maximum or restricted maximum likelihood estimation), significance tests (Wald-type, permutation, and likelihood-ratio tests), and methods for constructing confidence intervals (Wald-type and profile-likelihood intervals) for the scale coefficients of such models. When restricted maximum likelihood estimation was used, slightly closer to nominal rejection rates and narrower confidence intervals were obtained. The permutation test yielded type I error rates closest to the nominal level, whereas the likelihood-ratio test obtained the highest statistical power. In most scenarios, profile-likelihood intervals showed lower coverage probabilities than the Wald-type method but closer to the nominal 95% level. Finally, slightly higher rejection rates and coverage probabilities were obtained when a dichotomous moderator was examined rather than a continuous one. Despite the need to use some constraints on the parameter space for the scale coefficients and the possibility of non-convergence of some procedures that may affect the fitting of the specified models, location-scale models proved to be a valid and useful tool for modeling the heterogeneity parameter in meta-analysis.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"118"},"PeriodicalIF":4.6,"publicationDate":"2025-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11914364/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143647006","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Samply Stream API: The AI-enhanced method for real-time event data streaming.","authors":"Yury Shevchenko, Ulf-Dietrich Reips","doi":"10.3758/s13428-025-02634-1","DOIUrl":"10.3758/s13428-025-02634-1","url":null,"abstract":"<p><p>This manuscript introduces a novel method for conducting behavioral and social research by streaming real-time information to participants and manipulating content for experimental purposes via AI. We present an extension of the Samply software, which facilitates the integration of event-related data with mobile surveys and experiments. To assess the feasibility of this method, we conducted an experiment where news headlines were modified by a Chat-GPT algorithm and streamed to participants via the Samply Stream API and mobile push notifications. Feedback from participants indicated that most did not experience technical problems. There was no significant difference in readability across original, paraphrased, and misinformation-injected news conditions, with only 1.2% of all news items reported as unreadable. Participants reported significantly less familiarity with misinformation-injected news (84% unfamiliarity) compared to original and paraphrased news (73% unfamiliarity), suggesting successful manipulation of information without compromising readability. Dropout and non-response rates were comparable to those in other experience sampling studies. The streaming method offers significant potential for various applications, including public opinion research, healthcare, marketing, and environmental monitoring. By enabling the real-time collection of contextually relevant data, this method has the potential to enhance the external validity of behavioral research and provides a powerful tool for studying human behavior in naturalistic settings.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"119"},"PeriodicalIF":4.6,"publicationDate":"2025-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11914333/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143647012","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Revised network loadings.","authors":"Alexander P Christensen, Hudson Golino, Francisco J Abad, Luis Eduardo Garrido","doi":"10.3758/s13428-025-02640-3","DOIUrl":"10.3758/s13428-025-02640-3","url":null,"abstract":"<p><p>Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which aim to provide network models with a metric similar to factor loadings to assess measurement quality when the data are generated from a factor model. Our study revisits and refines the original network loadings to account for properties of (regularized) partial correlation networks, such as the reduction of partial correlation size as the number of variables increase, that were not considered previously. Using a simulation study, the revised network loadings demonstrated greater congruence with the simulated factor loadings across conditions relative to the original formulation. The simulation also evaluated how well correlations between factors can be captured by scores estimated with network loadings. The results show that not only can these network scores adequately estimate the simulated correlations between factors, they can do so without the need for rotation, a standard requirement for factor loadings. The consequence is that researchers do not need to choose a rotation with the revised network loadings, reducing the analytic degrees of freedom and eliminating this common source of variability in factor analysis. We discuss the interpretation of network loadings when data are believed to be generated from a network model and how they may fit into a network theory of measurement.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"114"},"PeriodicalIF":4.6,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11909041/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143633309","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Unraveling other-race face perception with GAN-based image reconstruction.","authors":"Moaz Shoura, Dirk B Walther, Adrian Nestor","doi":"10.3758/s13428-025-02636-z","DOIUrl":"https://doi.org/10.3758/s13428-025-02636-z","url":null,"abstract":"<p><p>The other-race effect (ORE) is the disadvantage of recognizing faces of another race than one's own. While its prevalence is behaviorally well documented, the representational basis of ORE remains unclear. This study employs StyleGAN2, a deep learning technique for generating photorealistic images to uncover face representations and to investigate ORE's representational basis. To this end, we collected pairwise visual similarity ratings with same- and other-race faces across East Asian and White participants exhibiting robust levels of ORE. Leveraging the significant overlap in representational similarity between the GAN's latent space and perceptual representations in human participants, we designed an image reconstruction approach aiming to reveal internal face representations from behavioral similarity data. This methodology yielded hyper-realistic depictions of face percepts, with reconstruction accuracy well above chance, as well as an accuracy advantage for same-race over other-race reconstructions, which mirrored ORE in both populations. Further, a comparison of reconstructions across participant race revealed a novel age bias, with other-race face reconstructions appearing younger than their same-race counterpart. Thus, our work proposes a new approach to exploiting the utility of GANs in image reconstruction and provides new avenues in the study of ORE.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"115"},"PeriodicalIF":4.6,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143633310","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Affective and non-affective psycholinguistic norms for 500 Chinese three-character idiomatic expressions.","authors":"Yanlu Zhong, Yimei Shao, Wei Yi","doi":"10.3758/s13428-025-02633-2","DOIUrl":"https://doi.org/10.3758/s13428-025-02633-2","url":null,"abstract":"<p><p>Multiword expressions, especially idiomatic expressions, convey rich emotional information. To explore the relationship between affective and non-affective psycholinguistic dimensions in the processing of idiomatic expressions, we collected normative data for two affective (i.e., valence and arousal) and four non-affective variables (i.e., familiarity, transparency, ambiguity, age of acquisition) for 500 Chinese three-character idiomatic expressions, based on responses from 418 native Chinese speakers. Our results showed a quadratic U-shaped relationship between valence and arousal, with neutral idiomatic expressions rated as less arousing than negative or positive ones. Furthermore, more transparent idiomatic expressions were more strongly valenced (either negative or positive) and more arousing. Idiomatic expressions more often used figuratively evoked more negative or stronger emotional responses. In addition, idiomatic expressions with greater negativity were acquired earlier but occurred less frequently. Regarding the non-affective variables, more transparent idiomatic expressions were perceived as more familiar and acquired earlier. Idiomatic expressions more often used figuratively, acquired earlier, or occurred more frequently were rated as more familiar. Lastly, idiomatic expressions that were more often used literally were more transparent and frequent. This study provides a large-scale database for examining the impact of affective and non-affective psycholinguistic factors on Chinese idiomatic expression processing, and it sheds light on the interrelated cognitive mechanisms underlying figurative language processing and emotion.</p>","PeriodicalId":8717,"journal":{"name":"Behavior Research Methods","volume":"57 4","pages":"116"},"PeriodicalIF":4.6,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143633307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}