Bayesian Two-Sample Hypothesis Testing Using the Uncertain Likelihood Ratio: Improving the Generalized Likelihood Ratio Test

Abstract

Two-sample hypothesis testing is a common practice in many fields of science, where the goal is to identify whether a set of observations and a set of training data are drawn from the same distribution. Traditionally, this is achieved using parametric and non-parametric frequentist tests, such as the Generalized Likelihood Ratio (GLR) test. However, these tests are not optimal in a Neyman-Pearson sense, especially when the number of observations and training samples are finite. Therefore, in this work, we study a parametric Bayesian test, called the Uncertain Likelihood Ratio (ULR) test, and compare its performance to the traditional GLR test. We establish that the ULR test is the optimal test for any number of samples when the parameters of the likelihood models are drawn from the true prior distribution. We then study an asymptotic form of the ULR test statistic and compare it against the GLR test statistic. As a byproduct of this analysis, we establish a new asymptotic optimality property for the GLR test when the parameters of the likelihood models are drawn from the Jeffreys prior. Furthermore, we analyze conditions under which the ULR test outperforms the GLR test, and include a numerical study to validate the results.