Psychological methods最新文献_第2页

Erroneous generalization-Exploring random error variance in reliability generalizations of psychological measurements.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-27 DOI: 10.1037/met0000740

Lukas J Beinhauer, Jens H Fünderich, Frank Renkewitz

{"title":"Erroneous generalization-Exploring random error variance in reliability generalizations of psychological measurements.","authors":"Lukas J Beinhauer, Jens H Fünderich, Frank Renkewitz","doi":"10.1037/met0000740","DOIUrl":"https://doi.org/10.1037/met0000740","url":null,"abstract":"Reliability generalization (RG) studies frequently interpret meta-analytic heterogeneity in score reliability as evidence of differences in an instrument's measurement quality across administrations. However, such interpretations ignore the fact that, under classical test theory, score reliability depends on two parameters: true score variance and error score variance. True score variance refers to the actual variation in the trait we aim to measure, while error score variance refers to nonsystematic variation arising in the observed, manifest variable. If the error score variance remains constant, variations in true score variance can result in heterogeneity in reliability coefficients. While this argument is not new, we argue that current approaches to addressing this issue in the RG literature are insufficient. Instead, we propose enriching an RG study with Boot-Err: Explicitly modeling the error score variance using bootstrapping and meta-analytic techniques. Through a comprehensive simulation scheme, we demonstrate that score reliability can vary while the measuring quality remains unaffected. The simulation also illustrates how explicitly modeling error score variances may improve inferences concerning random measurement error and under which conditions such enhancements occur. Furthermore, using openly available direct replication data, we show how explicitly modeling error score variance allows for an assessment to what extent measurement quality can be described as identical across administration sites. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143524271","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving the probability of reaching correct conclusions about congruence hypotheses: Integrating statistical equivalence testing into response surface analysis.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-24 DOI: 10.1037/met0000743

Sarah Humberg, Felix D Schönbrodt, Steffen Nestler

{"title":"Improving the probability of reaching correct conclusions about congruence hypotheses: Integrating statistical equivalence testing into response surface analysis.","authors":"Sarah Humberg, Felix D Schönbrodt, Steffen Nestler","doi":"10.1037/met0000743","DOIUrl":"https://doi.org/10.1037/met0000743","url":null,"abstract":"Many psychological theories imply that the degree of congruence between two variables (e.g., self-rated and objectively measured intelligence) is related to some psychological outcome (e.g., life satisfaction). Such congruence hypotheses can be tested with response surface analysis (RSA), in which a second-order polynomial regression model is estimated and suitably interpreted. Whereas several strategies exist for this interpretation, they all contain rationales that diminish the probability of drawing correct conclusions. For example, a frequently applied strategy involves calculating six auxiliary parameters from the estimated regression weights and accepting the congruence hypothesis if they satisfy certain conditions. In testing the conditions, a nonsignificant null-hypothesis test of some parameters is taken as evidence that the parameter is zero. This interpretation is formally inadmissible and adversely affects the probability of making correct decisions about the congruence hypothesis. We address this limitation of the six-parameter strategy and other RSA strategies by proposing that statistical equivalence testing (SET) be integrated into RSA. We compare the existing and new RSA strategies with a simulation study and find that the SET strategies are sensible alternatives to the existing strategies. We provide code templates for implementing the SET strategies. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143492805","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating statistical fit of confirmatory bifactor models: Updated recommendations and a review of current practice.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-20 DOI: 10.1037/met0000730

Sijia Li, Victoria Savalei

{"title":"Evaluating statistical fit of confirmatory bifactor models: Updated recommendations and a review of current practice.","authors":"Sijia Li, Victoria Savalei","doi":"10.1037/met0000730","DOIUrl":"https://doi.org/10.1037/met0000730","url":null,"abstract":"Confirmatory bifactor models have become very popular in psychological applications, but they are increasingly criticized for statistical pitfalls such as tendency to overfit, tendency to produce anomalous results, instability of solutions, and underidentification problems. In part to combat this state of affairs, many different reliability and dimensionality measures have been proposed to help researchers evaluate the quality of the obtained bifactor solution. However, in empirical practice, the evaluation of bifactor models is largely based on structural equation model fit indices. Other critical indicators of solution quality, such as patterns of general and group factor loadings, whether all estimates are interpretable, and values of reliability coefficients, are often not taken into account. In addition, in the methodological literature, some confusion exists about the appropriate interpretation and application of some bifactor reliability coefficients. In this article, we accomplish several goals. First, we review reliability coefficients for bifactor models and their correct interpretations, and we provide expectations for their values. Second, to help steer researchers away from structural equation model fit indices and to improve current practice, we provide a checklist for evaluating the statistical fit of bifactor models. Third, we evaluate the state of current practice by examining 96 empirical articles employing confirmatory bifactor models across different areas of psychology. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143468901","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Is a less wrong model always more useful? Methodological considerations for using ant colony optimization in measure development.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-20 DOI: 10.1037/met0000734

Yixiao Dong, Denis Dumas

{"title":"Is a less wrong model always more useful? Methodological considerations for using ant colony optimization in measure development.","authors":"Yixiao Dong, Denis Dumas","doi":"10.1037/met0000734","DOIUrl":"https://doi.org/10.1037/met0000734","url":null,"abstract":"With the advancement of artificial intelligence (AI), many AI-derived techniques have been adapted into psychological and behavioral science research, including measure development, which is a key task for psychometricians and methodologists. Ant colony optimization (ACO) is an AI-derived metaheuristic algorithm that has been integrated into the structural equation modeling framework to search for optimal (or near optimal) solutions. ACO-driven measurement modeling is an emerging method for constructing scales, but psychological researchers generally lack the necessary understanding of ACO-optimized models and outcome solutions. This article aims to investigate whether ACO solutions are indeed optimal and whether the optimized measurement models of ACO are always more psychologically useful compared to conventional ones built by human psychometricians. To work toward these goals, we highlight five essential methodological considerations for using ACO in measure development: (a) pursuing a local or global optimum, (b) avoiding a subjective optimum, (c) optimizing content validity, (d) bridging the gap between theory and model, and (e) recognizing limitations of unidirectionality. A joint data set containing item-level data from German (n = 297) and the United States (n = 334) samples was employed, and seven illustrative ACO analyses with various configurations were conducted to illustrate or facilitate the discussions of these considerations. We conclude that measurement solutions from the current ACO have not yet become optimal or close to optimal, and the optimized measurement models of ACO may be becoming more useful. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143468906","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Information theory, machine learning, and Bayesian networks in the analysis of dichotomous and Likert responses for questionnaire psychometric validation.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-17 DOI: 10.1037/met0000713

Matteo Orsoni, Mariagrazia Benassi, Marco Scutari

{"title":"Information theory, machine learning, and Bayesian networks in the analysis of dichotomous and Likert responses for questionnaire psychometric validation.","authors":"Matteo Orsoni, Mariagrazia Benassi, Marco Scutari","doi":"10.1037/met0000713","DOIUrl":"https://doi.org/10.1037/met0000713","url":null,"abstract":"Questionnaire validation is indispensable in psychology and medicine and is essential for understanding differences across diverse populations in the measured construct. While traditional latent factor models have long dominated psychometric validation, recent advancements have introduced alternative methodologies, such as the \"network framework.\" This study presents a pioneering approach integrating information theory, machine learning (ML), and Bayesian networks (BNs) into questionnaire validation. Our proposed framework considers psychological constructs as complex, causally interacting systems, bridging theories, and empirical hypotheses. We emphasize the crucial link between questionnaire items and theoretical frameworks, validated through the known-groups method for effective differentiation of clinical and nonclinical groups. Information theory measures such as Jensen-Shannon divergence distance and ML for item selection enhance discriminative power while contextually reducing respondent burden. BNs are employed to uncover conditional dependences between items, illuminating the intricate systems underlying psychological constructs. Through this integrated framework encompassing item selection, theory formulation, and construct validation stages, we empirically validate our method on two simulated data sets-one with dichotomous and the other with Likert-scale data-and a real data set. Our approach demonstrates effectiveness in standard questionnaire research and validation practices, providing insights into criterion validity, content validity, and construct validity of the instrument. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143441822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Robust Bayesian meta-regression: Model-averaged moderation analysis in the presence of publication bias.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-17 DOI: 10.1037/met0000737

František Bartoš, Maximilian Maier, T D Stanley, Eric-Jan Wagenmakers

{"title":"Robust Bayesian meta-regression: Model-averaged moderation analysis in the presence of publication bias.","authors":"František Bartoš, Maximilian Maier, T D Stanley, Eric-Jan Wagenmakers","doi":"10.1037/met0000737","DOIUrl":"10.1037/met0000737","url":null,"abstract":"Meta-regression is an essential meta-analytic tool for investigating sources of heterogeneity and assessing the impact of moderators. However, existing methods for meta-regression have limitations, such as inadequate consideration of model uncertainty and poor performance under publication bias. To overcome these limitations, we extend robust Bayesian meta-analysis (RoBMA) to meta-regression (RoBMA-regression). RoBMA-regression allows for moderator analyses while simultaneously taking into account the uncertainty about the presence and impact of other factors (i.e., the main effect, heterogeneity, publication bias, and other potential moderators). The methodology presents a coherent way of assessing the evidence for and against the presence of both continuous and categorical moderators. We further employ a Savage-Dickey density ratio test to quantify the evidence for and against the presence of the effect at different levels of categorical moderators. We illustrate RoBMA-regression in an empirical example and demonstrate its performance in a simulation study. We implemented the methodology in the RoBMA R package. Overall, RoBMA-regression presents researchers with a powerful and flexible tool for conducting robust and informative meta-regression analyses. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143441828","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Meta-analyzing nonpreregistered and preregistered studies.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-17 DOI: 10.1037/met0000719

Robbie C M van Aert

{"title":"Meta-analyzing nonpreregistered and preregistered studies.","authors":"Robbie C M van Aert","doi":"10.1037/met0000719","DOIUrl":"https://doi.org/10.1037/met0000719","url":null,"abstract":"Preregistration is gaining ground in psychology, and a consequence of this is that preregistered studies are more often included in meta-analyses. Preregistered studies likely mitigate the effect of publication bias in a meta-analysis, because preregistered studies can be located in the registries they were registered in even if they do not get published. However, current meta-analysis methods do not take into account that preregistered studies are less susceptible to publication bias. Traditional methods treat all studies as equivalent while meta-analytic conclusions can be improved by taking advantage of preregistered studies. The goal of this article is to introduce the hybrid extended meta-analysis (HYEMA) method that takes into account whether a study is preregistered or not and corrects for publication bias in only the nonpreregistered studies. The proposed method is applied to two meta-analyses on prominent effects in the psychological literature: the red-romance hypothesis and money priming. Applying HYEMA to these meta-analyses shows that the average effect size estimate is substantially closer to zero than the estimate of the random-effects meta-analysis model. Two simulation studies tailored to the two applications are also presented to illustrate the method's superior performance compared to the random-effects meta-analysis model and precision-effect test and precision-effect estimate with standard error when publication bias is present. Hence, I recommend to apply HYEMA as a sensitivity analysis if a mix of both preregistered and nonpreregistered studies are present in a meta-analysis. R code as well as a web application (https://rcmvanaert.shinyapps.io/HYEMA) have been developed and are described in the article to facilitate application of the method. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143441824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Fit indices are insensitive to multiple minor violations of perfect simple structure in confirmatory factor analysis.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-13 DOI: 10.1037/met0000718

Victoria Savalei, Muhua Huang

{"title":"Fit indices are insensitive to multiple minor violations of perfect simple structure in confirmatory factor analysis.","authors":"Victoria Savalei, Muhua Huang","doi":"10.1037/met0000718","DOIUrl":"https://doi.org/10.1037/met0000718","url":null,"abstract":"Classic confirmatory factor analysis (CFA) models are theoretically superior to exploratory factor analysis (EFA) models because they specify that each indicator only measures one factor. In contrast, in EFA, all loadings are permitted to be nonzero. In this article, we show that when fit to EFA structures and other models with many cross-loadings, classic CFA models often produce excellent fit. A key requirement for breaking this pattern is to have highly variable ratios of main loadings to corresponding cross-loadings in the true data-generating structure-and strongest misfit results when cross-loadings are of mixed sign. We show mathematically that EFA structures that are rotatable to a CFA representation are those where the main loadings and the cross-loadings are proportional for each group of indicators. With the help of a ShinyApp, we show that unless these proportionality constraints are violated severely in the true data structure, CFA models will fit well to most true models containing many cross-loadings by commonly accepted fit index cutoffs. We also show that fit indices are nonmonotone functions of the number of positive cross-loadings, and the relationship becomes monotone only when cross-loadings are of mixed sign. Overall, our findings indicate that good fit of a CFA model rules out that the true model is an EFA model with highly variable ratios of main and cross-loadings, but does not rule out most other plausible EFA structures. We discuss the implications of these findings. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143415023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Efficient design of cluster randomized trials and individually randomized group treatment trials.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-13 DOI: 10.1037/met0000727

Math J J M Candel, Gerard J P van Breukelen

{"title":"Efficient design of cluster randomized trials and individually randomized group treatment trials.","authors":"Math J J M Candel, Gerard J P van Breukelen","doi":"10.1037/met0000727","DOIUrl":"https://doi.org/10.1037/met0000727","url":null,"abstract":"For cluster randomized trials and individually randomized group treatment trials that compare two treatments on a continuous outcome, designs are presented that minimize the number of subjects or the amount of research budget, when aiming for a desired power level. These designs optimize the treatment-to-control allocation ratio of study participants but also optimize the choice between the number of clusters/groups versus the number of persons per cluster/group. Given that optimal designs require prior knowledge of parameters from the analysis model, which are often unknown during the design stage-especially outcome variances-maximin designs are introduced. These designs ensure a prespecified power level for plausible ranges of the unknown parameters and maximize power for the worst-case values of these parameters. The present study not only reviews but also extends the existing literature by deriving optimal and maximin designs when the number of clusters/groups are fixed because of practical constraints. How to calculate sample sizes in such practical designs and how much budget may be saved are illustrated for an empirical example. To facilitate sample size calculation for each of the variants of the maximin designs considered, an easy-to-use interactive R Shiny app has been developed and made available. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.6,"publicationDate":"2025-02-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143415022","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

How many factors to retain in exploratory factor analysis? A critical overview of factor retention methods.

IF 7.6 1区心理学

Psychological methods Pub Date : 2025-02-13 DOI: 10.1037/met0000733

David Goretzko

引用次数: 0