A SELECTIVE SURVEY OF SELECTIVE INFERENCE

Abstract

It is not difficult to find stories of a crisis in modern science, either in the popular press or in the scientific literature. There are likely multiple sources for this crisis. It is also well documented that one source of this crisis is the misuse of statistical methods in science, with the P -value receiving its fair share of criticism. It could be argued that this misuse of statistical methods is caused by a shift in how data is used in 21st century science compared to its use in the mid-20th century which presumed scientists had formal statistical hypotheses before collecting data. With the advent of sophisticated statistical software available to anybody this paradigm has been shifted to one in which scientists collect data first and ask questions later. 1 The new (?) scientific paradigm Figure 1: A simplified version of the scientific method. We are all familiar with a paradigm that does allow scientists to collect data first and ask questions later: the classical scientific method illustrated in Figure 1. A scientist collects data D, generates questions of interest Q(D), then collects fresh data D for confirmation and perhaps to discover additional questions of interest. The problem with this new paradigm is that it seeks to use D to answer these questions and may not have access to D. We pause here to note that Tukey used the term question rather than the more precise term hypothesis which statisticians might reasonably impute to be a statistical hypothesis. Given the computing capabilities of modern MSC2010: primary 62-02; secondary 62J15, 62J05, 62-07.