Testing homogeneity of effect sizes in pooling 2x2 contingency tables from multiple studies: a comparison of methods

Abstract Meta-analysis is a statistical methodology that combines the outcomes of several independent studies. Either a fixed-effects or a random-effects model is used to combine the results of the independent studies. The choice depends on whether the outcomes can be considered homogeneous or not. Several methods have been developed to test the homogeneity of effect sizes. Although many of them have been compared already, they have not been studied in more realistic practical settings where individuals all have their own risk of an event and a complete overview is lacking. In this article, we investigate the performance of 10 statistical methods to test homogeneity of a binary exposure on a binary clinical outcome, using an extensive simulation study and a real life meta-analysis. We evaluated the Type I error and the statistical power. The fixed-effects regression model for treatment and study interaction was found to be a slightly liberal test, while the Q-statistic and the Bliss statistic can be considered conservative tests. The random-effects regression model, the Peto statistic and the I2 perform rather poorly compared to the other methods. All chi-square tests that are based on the calculation of a specific odds ratio and the fixed-effects logistic regression analysis perform best. Among these chi-square tests, we recommend the Breslow–Day test, but only for convenience purposes, since it is available in most statistical software packages and the other tests are not.