Comments on “Single paper meta-analysis is unavoidable”

Abstract

The method dialogue by Blakeley McShane and Ulf Böckenholt (2025) provides a strong critique of the way behavioral scholars have analyzed and presented their findings. The initial document was sent to four established scholars who agreed to provide open collaborative guidance to the authors. Following the accepted revised document, three of the four collaborating authors and others provide their final reactions below. Christian Wheeler summarizes and clarifies the recommendations of McShane and Böckenholt and assesses their effect on current research practices. He views that Single Paper Meta-analysis (SPM) improves theory development in the social sciences by enabling researchers to better understand the inherent variability in empirical relationships. He believes the focus on point and range effect estimates can beneficially move research away from a dichotomized world to a theoretically richer one that articulates the credibility and magnitude of effects. John Lynch acknowledges the costly current practice in which authors intuitively summarize the aggregate evidence across multiple studies. He advocates a shift in focus from estimates of probabilities of null hypotheses towards meaningful metrics specifying the change in the magnitude of a dependent variable. He also counters the claim that SPM might be fraudulently used to promote false conclusions by asserting that it encourages disclosure of evidence and promotes better communication of the heterogeneity of results from different studies. The null hypothesis of zero effect is never true, and the goal of meta-analysis should never be to test some null hypothesis of zero average effect. Wegener, Pek and Matthews celebrate the philosophical value of meta-analyses for evaluating multi-study empirical cases. They note that, especially in single paper settings, the hypothetical alternative of a true null effect remains a relevant hypothesis to reject, and SPM provides a more principled and accurate means of such assessments. SPM is better than human intuition at generating estimates of the aggregate effect across studies. As evidence, they show that researchers’ intuition influences perceived credibility of the aggregate effect across studies in ways that would not be justified by SPM. By providing a compact description of aggregate effects, SPM and related techniques can provide an improved assessment of the statistical reliability of a given effect.