Scalable Empirical Bayes Inference and Bayesian Sensitivity Analysis.

Abstract

Consider a Bayesian setup in which we observe $Y$ , whose distribution depends on a parameter $\theta$ , that is, $Y\mid \theta ~{\pi }_{Y\mid \theta }$ . The parameter $\theta$ is unknown and treated as random, and a prior distribution chosen from some parametric family $\left({\pi }_{\theta }(\cdot ;h),h\in \mathscr{H}\right)$ , is to be placed on it. For the subjective Bayesian there is a single prior in the family which represents his or her beliefs about $\theta$ , but determination of this prior is very often extremely difficult. In the empirical Bayes approach, the latent distribution on $\theta$ is estimated from the data. This is usually done by choosing the value of the hyperparameter $h$ that maximizes some criterion. Arguably the most common way of doing this is to let $m\left(h\right)$ be the marginal likelihood of $h$ , that is, $m\left(h\right)=\int {\pi }_{Y\mid \theta }{v}_h\left(\theta \right)d\theta$ , and choose the value of $h$ that maximizes $m(\cdot )$ . Unfortunately, except for a handful of textbook examples, analytic evaluation of $arg{max}_{h}m\left(h\right)$ is not feasible. The purpose of this paper is two-fold. First, we review the literature on estimating it and find that the most commonly used procedures are either potentially highly inaccurate or don't scale well with the dimension of $h$ , the dimension of $\theta$ , or both. Second, we present a method for estimating $arg{max}_{h}m\left(h\right)$ , based on Markov chain Monte Carlo, that applies very generally and scales well with dimension. Let $g$ be a real-valued function of $\theta$ , and let $I\left(h\right)$ be the posterior expectation of $g\left(\theta \right)$ when the prior is $v_h$ . As a byproduct of our approach, we show how to obtain point estimates and globally-valid confidence bands for the family $I\left(h\right)$ , $h\in \mathscr{H}$ . To illustrate the scope of our methodology we provide three detailed examples, having different characters.