Efficient statistics: Extracting information from IID observations

Abstract

In this paper, we study how information can be conveyed through a noisy channel and extracted efficiently, under the scenarios and applications, where the observing order of the symbols does not carry any useful information. In such cases, the information-carrying objects are the empirical distributions of the transmitted and received symbol sequences. We develop a local geometric structure and a new coordinate system for the space of distributions. With this approach, we can decompose the computation of the posterior distribution of the data into a sequence of score functions, with decreasing information volumes. Thus, when our goal is not to recover the entire data, but only to detect certain features of the data, we only need to compute the first few scores, which greatly simplifies the problem. We demonstrate the use of our technique with some image processing examples based on graphical models.