All is well that replicates well: The replicability of reported moderation and interaction effects in leading organizational sciences journals.

IF 9.4 1区心理学 Q1 MANAGEMENT

Journal of Applied Psychology Pub Date : 2024-10-01 Epub Date: 2024-05-09 DOI:10.1037/apl0001197

Marcus Crede, Lukas K Sotola

{"title":"All is well that replicates well: The replicability of reported moderation and interaction effects in leading organizational sciences journals.","authors":"Marcus Crede, Lukas K Sotola","doi":"10.1037/apl0001197","DOIUrl":null,"url":null,"abstract":"We examine 244 independent tests of interaction effects published in recent issues of four leading journals in the organizational sciences in order to estimate the replicability of reported statistically significant interaction effects. A z-curve analysis (Brunner & Schimmack, 2020) of the distribution of p values indicates an estimated replicability of 37%, although this figure varied somewhat across the four journals. We also find that none of the coded studies reported having conducted a priori power analyses and that only one reported having preregistered their hypotheses-despite longstanding exhortations for researchers to plan their studies to have adequate power and to engage in open science practices. Our results suggest that moderation results that have been reported in these leading journals fail to meet the methodological and statistical burden that would lead us to recommend that scientists and practitioners rely on these findings to inform their research and practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).","PeriodicalId":15135,"journal":{"name":"Journal of Applied Psychology","volume":" ","pages":"1659-1667"},"PeriodicalIF":9.4000,"publicationDate":"2024-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Applied Psychology","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1037/apl0001197","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/5/9 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"MANAGEMENT","Score":null,"Total":0}

引用次数: 0

Abstract

We examine 244 independent tests of interaction effects published in recent issues of four leading journals in the organizational sciences in order to estimate the replicability of reported statistically significant interaction effects. A z-curve analysis (Brunner & Schimmack, 2020) of the distribution of p values indicates an estimated replicability of 37%, although this figure varied somewhat across the four journals. We also find that none of the coded studies reported having conducted a priori power analyses and that only one reported having preregistered their hypotheses-despite longstanding exhortations for researchers to plan their studies to have adequate power and to engage in open science practices. Our results suggest that moderation results that have been reported in these leading journals fail to meet the methodological and statistical burden that would lead us to recommend that scientists and practitioners rely on these findings to inform their research and practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

查看原文本刊更多论文

复制得好，一切都好：主要组织科学期刊中报告的调节和交互效应的可复制性。

我们研究了组织科学领域四种主要期刊近期发表的 244 篇交互效应独立测试文章，以估算所报告的具有统计意义的交互效应的可复制性。对 p 值分布进行的 z 曲线分析（Brunner & Schimmack，2020 年）表明，估计可复制性为 37%，但这一数字在四种期刊中略有不同。我们还发现，没有一项编码研究报告进行了先验功率分析，只有一项报告预先注册了假设--尽管研究人员长期以来一直在呼吁规划研究以获得足够的功率并参与开放科学实践。我们的研究结果表明，这些主要期刊所报道的调节结果未能满足方法学和统计学的要求，因此我们建议科学家和从业人员依靠这些结果来指导他们的研究和实践。(PsycInfo Database Record (c) 2024 APA, 版权所有）。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Applied Psychology Multiple-

CiteScore

17.60

自引率

6.10%

发文量

175

期刊介绍： The Journal of Applied Psychology® focuses on publishing original investigations that contribute new knowledge and understanding to fields of applied psychology (excluding clinical and applied experimental or human factors, which are better suited for other APA journals). The journal primarily considers empirical and theoretical investigations that enhance understanding of cognitive, motivational, affective, and behavioral psychological phenomena in work and organizational settings. These phenomena can occur at individual, group, organizational, or cultural levels, and in various work settings such as business, education, training, health, service, government, or military institutions. The journal welcomes submissions from both public and private sector organizations, for-profit or nonprofit. It publishes several types of articles, including: 1.Rigorously conducted empirical investigations that expand conceptual understanding (original investigations or meta-analyses). 2.Theory development articles and integrative conceptual reviews that synthesize literature and generate new theories on psychological phenomena to stimulate novel research. 3.Rigorously conducted qualitative research on phenomena that are challenging to capture with quantitative methods or require inductive theory building.