Sample size and geographical region predict effect heterogeneity in psychotherapy research for depression: a meta-epidemiological study

Abstract

Objectives

Depression is a very prevalent and burdensome disease, for which the efficacy of psychological treatments has been extensively studied in randomized controlled trials (RCTs). Meta-analytic evidence in this field is often heavily limited due to heterogeneity, meaning a broad dispersion of true treatment effects and high uncertainty when predicting future outcomes. Causes for this heterogeneity are largely unclear, and cannot be directly examined using conventional meta-analytic methods. Using newly introduced location-scale models, this study is the first to examine direct predictors of between-study heterogeneity in depression psychotherapy trials.

Study Design and Setting

We used a large meta-analytic database containing RCTs on the efficacy of depression psychotherapy. We included studies in all age groups, comparing psychotherapy to control conditions. Risk of bias (RoB) was assessed with the “Cochrane Collaboration Risk of Bias Tool” (Version 1). Univariate analyses were used to explore associations of study-level variables with treatment effect heterogeneity, and multimodel selection to investigate the predictive effect of all variables simultaneously.

Results

We included 539 RCTs with 607 comparisons, with 35% showing low overall RoB. Higher heterogeneity was found in studies with high RoB and lower sample sizes; heterogeneity varied depending on the geographical region where trials were conducted. Based on multimodel selection, the most important predictors of effect heterogeneity were geographical region, baseline sample size, and RoB. These predictors were also significant after model averaging.

Conclusion

Our study shows that several study-level variables predict the heterogeneity of treatment effects in psychotherapy research, and thus their predictability across different contexts. To enhance the robustness of pooled effects, meta-analysts may consider restricting their synthesis to methodologically rigorous studies only. Our findings also indicate that the assumption of constant heterogeneity in “traditional” random-effects analyses might often be violated, making sensitivity analyses imperative.