Statistical fragility in the meta-analytic evidence base of animal cognition: A meta-meta-analytic reappraisal.

Abstract

How reliable is the meta-analytic evidence base in animal cognition? We conducted a pre-registered second-order meta-analysis of 28 published meta-analytical papers, comprising 48 eligible models and 5,740 effect-size observations from primary studies. We assessed statistical power, Type M magnitude error, and Type S sign error at both primary-study and meta-analysis levels. Because these quantities depend on the assumed underlying effect, we compared results based on uncorrected pooled effects, a regression-based bias correction, and a bias-robust FE+VCV sensitivity analysis. At the primary-study level, estimated power was low under all three empirically estimated effect proxies: 17.4%, 9.0%, and 13.4%, respectively. Type M and Type S errors were also sensitive to the assumed effect and correction method. Power increased when larger externally specified effects were assumed, confirming that these quantities are conditional on the assumed effect. Meta-analysis-level results were generally more favourable but sensitive to analytical choices. The evidence base was concentrated in experimental rodent studies of learning and memory, limiting field-wide generalisation. These findings support improving study design and reporting within the represented literature while expanding meta-analytic coverage across taxa and cognitive domains.