Compression and predictive distributions for large alphabet i.i.d and Markov models

Abstract

This paper considers coding and predicting sequences of random variables generated from a large alphabet. We start from the i.i.d model and propose a simple coding distribution formulated by a product of tilted Poisson distributions which achieves close to optimal performance. Then we extend to Markov models, and in particular, tree sources. A context tree based algorithm is designed according to the frequency of various contexts in the data. It is a greedy algorithm which seeks for the greatest savings in codelength when constructing the tree. Compression and prediction of individual counts associated with the contexts again uses a product of tilted Poisson distributions. Implementing this method on a Chinese novel, about 20.56% savings in codelength is achieved compared to the i.i.d model.