Crowdsourcing multiverse analyses to explore the impact of different data-processing and analysis decisions: A tutorial.

IF 7.8 1区心理学 Q1 PSYCHOLOGY, MULTIDISCIPLINARY

Psychological methods Pub Date : 2025-09-18 DOI:10.1037/met0000770

Tom Heyman,Ekaterina Pronizius,Savannah C Lewis,Oguz A Acar,Matúš Adamkovič,Ettore Ambrosini,Jan Antfolk,Krystian Barzykowski,Ernest Baskin,Carlota Batres,Leanne Boucher,Jordane Boudesseul,Eduard Brandstätter,W Matthew Collins,Dušica Filipović Ðurđević,Ciara Egan,Vanessa Era,Paulo Ferreira,Chiara Fini,Patricia Garrido-Vásquez,Hendrik Godbersen,Pablo Gomez,Aurelien Graton,Necdet Gurkan,Zhiran He,Dave C Johnson,Pavol Kačmár,Chris Koch,Marta Kowal,Tomas Kratochvil,Marco Marelli,Fernando Marmolejo-Ramos,Martín Martínez,Alan Mattiassi,Nicholas P Maxwell,Maria Montefinese,Coby Morvinski,Maital Neta,Yngwie A Nielsen,Sebastian Ocklenburg,Jaš Onič,Marietta Papadatou-Pastou,Adam J Parker,Mariola Paruzel-Czachura,Yuri G Pavlov,Manuel Perea,Gerit Pfuhl,Tanja C Roembke,Jan P Röer,Timo B Roettger,Susana Ruiz-Fernandez,Kathleen Schmidt,Cynthia S Q Siew,Christian K Tamnes,Jack E Taylor,Rémi Thériault,José L Ulloa,Miguel A Vadillo,Michael E W Varnum,Martin R Vasilev,Steven Verheyen,Giada Viviani,Sebastian Wallot,Yuki Yamada,Yueyuan Zheng,Erin M Buchanan

{"title":"Crowdsourcing multiverse analyses to explore the impact of different data-processing and analysis decisions: A tutorial.","authors":"Tom Heyman,Ekaterina Pronizius,Savannah C Lewis,Oguz A Acar,Matúš Adamkovič,Ettore Ambrosini,Jan Antfolk,Krystian Barzykowski,Ernest Baskin,Carlota Batres,Leanne Boucher,Jordane Boudesseul,Eduard Brandstätter,W Matthew Collins,Dušica Filipović Ðurđević,Ciara Egan,Vanessa Era,Paulo Ferreira,Chiara Fini,Patricia Garrido-Vásquez,Hendrik Godbersen,Pablo Gomez,Aurelien Graton,Necdet Gurkan,Zhiran He,Dave C Johnson,Pavol Kačmár,Chris Koch,Marta Kowal,Tomas Kratochvil,Marco Marelli,Fernando Marmolejo-Ramos,Martín Martínez,Alan Mattiassi,Nicholas P Maxwell,Maria Montefinese,Coby Morvinski,Maital Neta,Yngwie A Nielsen,Sebastian Ocklenburg,Jaš Onič,Marietta Papadatou-Pastou,Adam J Parker,Mariola Paruzel-Czachura,Yuri G Pavlov,Manuel Perea,Gerit Pfuhl,Tanja C Roembke,Jan P Röer,Timo B Roettger,Susana Ruiz-Fernandez,Kathleen Schmidt,Cynthia S Q Siew,Christian K Tamnes,Jack E Taylor,Rémi Thériault,José L Ulloa,Miguel A Vadillo,Michael E W Varnum,Martin R Vasilev,Steven Verheyen,Giada Viviani,Sebastian Wallot,Yuki Yamada,Yueyuan Zheng,Erin M Buchanan","doi":"10.1037/met0000770","DOIUrl":null,"url":null,"abstract":"When processing and analyzing empirical data, researchers regularly face choices that may appear arbitrary (e.g., how to define and handle outliers). If one chooses to exclusively focus on a particular option and conduct a single analysis, its outcome might be of limited utility. That is, one remains agnostic regarding the generalizability of the results, because plausible alternative paths remain unexplored. A multiverse analysis offers a solution to this issue by exploring the various choices pertaining to data-processing and/or model building, and examining their impact on the conclusion of a study. However, even though multiverse analyses are arguably less susceptible to biases compared to the typical single-pathway approach, it is still possible to selectively add or omit pathways. To address this issue, we outline a novel, more principled approach to conducting multiverse analyses through crowdsourcing. The approach is detailed in a step-by-step tutorial to facilitate its implementation. We also provide a worked-out illustration featuring the Semantic Priming Across Many Languages project, thereby demonstrating its feasibility and its ability to increase objectivity and transparency. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"1 1","pages":""},"PeriodicalIF":7.8000,"publicationDate":"2025-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Psychological methods","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1037/met0000770","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

When processing and analyzing empirical data, researchers regularly face choices that may appear arbitrary (e.g., how to define and handle outliers). If one chooses to exclusively focus on a particular option and conduct a single analysis, its outcome might be of limited utility. That is, one remains agnostic regarding the generalizability of the results, because plausible alternative paths remain unexplored. A multiverse analysis offers a solution to this issue by exploring the various choices pertaining to data-processing and/or model building, and examining their impact on the conclusion of a study. However, even though multiverse analyses are arguably less susceptible to biases compared to the typical single-pathway approach, it is still possible to selectively add or omit pathways. To address this issue, we outline a novel, more principled approach to conducting multiverse analyses through crowdsourcing. The approach is detailed in a step-by-step tutorial to facilitate its implementation. We also provide a worked-out illustration featuring the Semantic Priming Across Many Languages project, thereby demonstrating its feasibility and its ability to increase objectivity and transparency. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

查看原文本刊更多论文

众包多元宇宙分析，探索不同数据处理和分析决策的影响：教程。

在处理和分析经验数据时，研究人员经常面临可能显得武断的选择（例如，如何定义和处理异常值）。如果一个人选择专门关注一个特定的选项，并进行单一的分析，其结果可能是有限的效用。也就是说，对于结果的普遍性，人们仍然是不可知论者，因为合理的替代途径仍然没有被探索。多元宇宙分析通过探索与数据处理和/或模型构建相关的各种选择，并检查它们对研究结论的影响，为这个问题提供了解决方案。然而，尽管与典型的单路径方法相比，多元宇宙分析可以说不太容易受到偏差的影响，但仍然有可能选择性地添加或省略路径。为了解决这个问题，我们概述了一种新颖的、更有原则的方法，通过众包来进行多元宇宙分析。该方法将在一个循序渐进的教程中详细介绍，以促进其实现。我们还提供了一个针对跨多种语言语义启动项目的详细说明，从而展示了其可行性及其增加客观性和透明度的能力。（PsycInfo Database Record (c) 2025 APA，版权所有）。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Psychological methods PSYCHOLOGY, MULTIDISCIPLINARY-

CiteScore

13.10

自引率

7.10%

发文量

159

期刊介绍： Psychological Methods is devoted to the development and dissemination of methods for collecting, analyzing, understanding, and interpreting psychological data. Its purpose is the dissemination of innovations in research design, measurement, methodology, and quantitative and qualitative analysis to the psychological community; its further purpose is to promote effective communication about related substantive and methodological issues. The audience is expected to be diverse and to include those who develop new procedures, those who are responsible for undergraduate and graduate training in design, measurement, and statistics, as well as those who employ those procedures in research.