Commentary to MARP: how to increase the robustness of survey studies

Abstract

This comment aims at one attribute that leaves room for improvement of the generally well-thought-out many-analysts-religion-project (MARP) approach (Hoogeveen, et al., 2022): All teams used only one data set. We present our analyses based on an extended database, referring to literature on replicability and the relationship between religiosity and well-being before ending with recommendations for future projects of this kind. The MARP approach will give new insights into the challenge of replicability of research fi ndings inherent to every project that is carried out by only one researcher or research group. What remains is the problem of drawing reliable conclusions based on only one data set. One data set might always be prone to selection bias and it might lack representativity as well as potentially important variables that were not collected. In order to argue for a robust e ff ect between well-being and religiosity, the e ff ect should be still measurable when interchanging the collected data with other survey data or when adding additional variables, including from other data sets, in order to avoid omitted variable bias. In economics, Clemens (2017) presents a classi fi cation for replication studies, which distinguishes among four types of replication and robustness checks: veri fi cation, reproduction, reanalysis, and extension (see Table 1). If we classify the MARP approach according to this classi fi cation scheme, we would consider it a robustness check of the reanalysis type. The participating researchers used the same population, although they speci fi ed their samples and analytical models di ff erently. To achieve a replication of the studies within MARP, any researcher could use the published analysis scripts and repeat the analyses with the same or another sample.