A decade on: Reflecting on the limitations of the first meta-analysis of the Implicit Relational Assessment Procedure's (IRAP) criterion validity in the clinical domain

Abstract

Hussey (in press) recently conducted a detailed critical reanalysis of Vahey, Nicholson and Barnes-Holmes’ (2015) meta-analysis. Its stated purpose was to (a) examine the extent to which Vahey et al.’s (2015) meta-analysis contains errors; and (b) to test how computationally reproducible it is by current standards of best practice. Hussey identified a small number of minor numerical errors, but crucially was unable to exactly replicate the original meta-effect of $\bar{r}$ = .45. Six different variations of the meta-analysis reported by Vahey et al. were used and obtained meta-effects that deviated from the original by Δ $\bar{r}$ = .01-.02. Hussey also reported corresponding 95% credibility intervals that were all of zero width. These discrepancies prompted the present authors to conduct a detailed audit of the original meta-analysis. This revealed one minor transposing error in addition to three identified by Hussey. Once corrected this resulted in a marginally increased Hunter and Schmidt meta-analytic effect of $\bar{r}$ = .46 without a credibility interval, and a Hedges-Vevea meta-effect of $\bar{r}$ = .47 with 95% confidence interval (.40, .54). This correction was too small to have any bearing on Vahey et al.’s supplementary analyses regarding publication bias or statistical power. Vahey et al. contained a much lower proportion of transposing errors than is typical of meta-analyses even still (cf. Kadlec, Sainani, & Nimphius, 2023; Lakens et al., 2016; Lakens et al., 2017). Nonetheless, Hussey highlighted important ambiguities about the theoretical and practical meaning of the meta-effect reported by Vahey et al. We clarify our position on these matters in summary, and in so doing explain why we believe that the wider IRAP literature would undoubtedly benefit from increased adoption of contemporary open science standards.