A Geometric Characterization of Fisher Information from Quantized Samples with Applications to Distributed Statistical Estimation

Abstract

Consider the Fisher information for estimating a vector $\theta \in \mathbb {R}^{d}$ from the quantized version of a statistical sample $X \sim f(x|\theta)$. Let M be a k-bit quantization of $X.$ We provide a geometric characterization of the trace of the Fisher information matrix $I_{M}(\theta)$ in terms of the score function $S_{\theta }(X)$. When $k=1$, we exactly solve the extremal problem of maximizing this geometric quantity for the Gaussian location model, which allows us to conclude that in this model, a half-space quantization is the one-bit quantization that maximizes $Tr(I_{M}(\theta))$. Under assumptions on the tail of the distribution of $S_{\theta }(X)$ projected onto any unit vector in $\mathbb {R}^{d}$, we give upper bounds demonstrating how $Tr(I_{M}(\theta))$ can scale with k. We apply these results to find lower bounds on the minimax risk of estimating $\theta $ from multiple quantized samples of X, for example in a distributed setting where the samples are distributed across multiple nodes and each node has a total budget of k-bits to communicate its sample to a centralized estimator. Our bounds apply in a unified way to many common statistical models including the Gaussian location model and discrete distribution estimation, and they recover and generalize existing results in the literature with simpler and more transparent proofs.