Avoiding common mistakes in meta-analysis: Understanding the distinct roles of Q, I-squared, tau-squared, and the prediction interval in reporting heterogeneity

Abstract

In any meta-analysis, it is critically important to report the dispersion in effects as well as the mean effect. If an intervention has a moderate clinical impact on average we also need to know if the impact is moderate for all relevant populations, or if it varies from trivial in some to major in others. Or indeed, if the intervention is beneficial in some cases but harmful in others. Researchers typically report a series of statistics such as the Q-value, the p-value, and I², which are intended to address this issue. Often, they use these statistics to classify the heterogeneity as being low, moderate, or high and then use these classifications when considering the potential utility of the intervention. While this practice is ubiquitous, it is nevertheless incorrect. The statistics mentioned above do not actually tell us how much the effect size varies. Classifications of heterogeneity based on these statistics are uninformative at best, and often misleading. My goal in this paper is to explain what these statistics do tell us, and that none of them tells us how much the effect size varies. Then I will introduce the prediction interval, the statistic that does tell us how much the effect size varies, and that addresses the question we have in mind when we ask about heterogeneity. This paper is adapted from a chapter in “Common Mistakes in Meta-Analysis and How to Avoid Them.” A free PDF of the book is available at https://www.Meta-Analysis.com/rsm.