How to quantify between-study heterogeneity in single-arm evidence synthesis?-It depends!

Background: Random-effects meta-analysis models account for between-study heterogeneity by estimating and incorporating the heterogeneity variance parameter ${\tau }^2$ . Numerous estimators for ${\tau }^2$ have been proposed, but no widely accepted guidance exists on when to best use which heterogeneity variance estimator. Especially in the context of single-arm observational studies, studies with unique challenges, such as outcome measure variability, sparse data, and high methodological heterogeneity, systematic evaluations and comparisons of the various heterogeneity variance estimators are lacking. This study investigates the advantages of different heterogeneity variance estimators for typical single-arm meta-analysis scenarios through comprehensive simulations in a neutral comparison study setting and with an empirical application in pediatrics.

Methods: We compared seven heterogeneity variance estimators for random-effects meta-analysis. The estimators were selected on the basis of methodological diversity and availability and were evaluated both empirically and in a simulation study. We simulated typical meta-analysis scenarios for continuous and binary outcomes in a single-arm meta-analysis setting. Through a non-systematic literature review, we assessed which heterogeneity variance estimators are currently used in high-ranked journals, and evaluated their reporting quality.

Results: Our simulation study showed that all evaluated heterogeneity estimators were imprecise and often failed to estimate the true amount of heterogeneity. The estimation is particularly imprecise in situations where the meta-analysis contained few studies or when the binary outcomes included rare events. Moreover, we discovered that most heterogeneity variance estimators produce zero heterogeneity estimates under all simulated conditions, even though heterogeneity was present. The estimated overall effect was found to be relatively robust to different estimators in the empirical application and in our simulation study. However, the prediction intervals for the overall effect vary depending on the estimator chosen.

Conclusions: Although different heterogeneity variance estimators produce substantially different heterogeneity variance estimates, too little attention is paid to selecting a suitable heterogeneity variance estimator in single-arm evidence synthesis. Based on our literature review, we conclude that the awareness of different heterogeneity variance estimators and their properties needs to be strengthened in practice. Given that it is rarely appropriate to rely on a single heterogeneity variance estimator, we suggest careful consideration and evaluation of a wider range of plausible estimators in a sensitivity analysis before drawing a final conclusion about the meta-analysis results.