A Chain as Strong as Its Strongest Link? Understanding the Causes and Consequences of Biases Arising from Selective Analysis and Reporting of Research Results

Abstract

This is a big problem in applied statistics in general and education policy in particular. For example, the respected economist James Heckman and his colleagues have claimed (Garcia et al., 2016) that, for a certain early childhood intervention, “The overall rate of return is 13.7% per annum, and the benefit/cost ratio is 7.3.” That quoted sentence has two serious problems related to uncertainty and selection bias. To start with, it is an elementary but important error to have written that “the overall rate of return is . . . the benefit/cost ratio is . . .”; each instance of the word “is” in that sentence should immediately have been followed by “estimated at.” The next step is to recognize that selection on statistical significance induces biases in these estimates— and, given the small samples, high variabilities, and researcher degrees of freedom in the studies where the estimates came from, the biases could be huge (Gelman, 2017a). The report with its bold claims neither acknowledged this bias nor made any attempt to assess its magnitude.