Rethinking Probability of Success as Bayes Utility

Abstract

In the hybrid frequentist-Bayesian approach, the probability of success (PoS) of a trial is the expected value of the traditional power function of a test with respect to a design prior assigned to the parameter under scrutiny. However, this definition is not univocal and some of the proposals do not lack of potential drawbacks. These problems are related to the fact that such definitions are all based on the probability of rejecting the null hypothesis rather than on the probability of choosing the correct hypothesis, be it the null or the alternative. In this article, we propose a unifying, decision-theoretic approach that yields a new definition of PoS as the expected utility of the trial (u-PoS), that is, as the expected probability of making the correct choice between the two hypotheses. This proposal shows a conceptual advantage over previous definitions of PoS; moreover, it produces smaller optimal sample sizes whenever the design prior assigns positive probability to the null hypothesis.