So is it better than something else? Using the results of a random-effects meta-analysis to characterize the magnitude of an effect size as a percentile.

Abstract

The characterization of an effect size is best made in reference to effect sizes found in the literature. A random-effects meta-analysis is the systematic synthesis of related effects from across a literature, producing an estimate of the distribution of effects in the population. We propose using the estimated mean and variance from a random-effects meta-analysis to inform the characterization of an observed effect size. The percentile of an observed effect size within the estimated distribution of population effects can describe the magnitude of the observed effect. Because there is uncertainty in the population estimates, we propose using the prediction distribution (used frequently to estimate the prediction interval in a meta-analysis) to serve as the reference distribution when characterizing an effect size. Doing so, the percentile of an observed effect and the limits of the effect size's 95% confidence interval within the prediction distribution are calculated. With numerous meta-analyses available including various outcomes and contexts, the presented method can be useful to many researchers and practitioners. We demonstrate the application of an easy-to-use Excel worksheet to automate these percentile calculations. We follow this with a simulation study evaluating the method's performance over a range of conditions. Recommendations (and cautions) for meta-analysts and researchers conducting a single study are provided. (PsycInfo Database Record (c) 2024 APA, all rights reserved).