When Does it Make Sense to Perform a Meta‐Analysis?

In the early days of meta-analysis (at least in its current incarnation) Robert Rosenthal was asked if it makes sense to perform a meta-analysis, given that the studies differ in various ways, and the analysis amounts to combining apples and oranges. Rosenthal answered that combining apples and oranges makes sense if your goal is to produce a fruit salad. The goal of a meta-analysis is only rarely to synthesize data from a set of identical studies. Almost invariably, the goal is to broaden the base of studies in some way, expand the question, and study the pattern of answers. The question of whether it makes sense to perform a meta-analysis, and the question of what kinds of studies to include, must be asked and answered in the context of specific goals. The ability to combine data from different studies to estimate the common effect (or mean effect), continues to be an important function of meta-analysis. However, it is not the only function. The goal of some syntheses will be to report the summary effect, but the goal of other syntheses will be to assess the dispersion as well as the mean effect, and the goal of others will be to focus on the dispersion exclusively. For example, suppose that we are looking at the impact of a teaching intervention on student performance. Does it make sense to include studies that measured verbal skills and also studies that measured math skills? If our goal is to assess the impact on performance in general, then the answer is Yes. If our goal is to assess the impact on verbal skills alone, then the answer is No. Does it make sense to include studies