Bayesian Estimation: Information-Theoretic Analysis and Applications

Abstract

It is known that the optimal redundancy of a source code behaves like one half the dimension of the parameter times the log of the sample size. We give an asymptotic expression for the redundancy which is valid for smooth parametric families of distributions equipped with a prior. Important terms include one half the logarithm of the determinant of the Fisher information matrix, minus the logarithm of the prior density and a constant arising from the mean of a Chi-square distribution. The dominant terms arise from an integration by Laplace's method. Our formula can be integrated with respect to the prior distribution on the parameter, under some conditions, so as to give the average redundancy and the minimax redundancy. The minimax code uses Jeff reys' prior. The same expansion has implications for channel coding: Consider channels which have a continuous d-dimensional input alphabet and a k-dimensional output alphabet (where the coordinates of the output are conditionally independent of the input). A message for this channel is Cooperatively encoded by d transmitters and cooperatively decoded by n receivers. For a large number of receivers the mutual information behaves like (d/ 2) logk , that is, one half the number of transmitters times the log of the number of receivers. From the estimation standpoint we have approximated three forms of the cumulative risk under relative entropy loss. Our asymptotic expansions give the risk, the Bayes risk, and the minimax risk. The cumulative risk of the Bayes estimator occurs naturally as the error exponent in a hypothesis test. It also occurs naturally in proving that the standardized posterior converges to a normal.