Single letter characterization of average-case strong redundancy of compressing memoryless sequences

Abstract

We obtain a condition that is both necessary and sufficient to characterize strong universal compressibility (in the average sense) of sequences generated by iid sampling from a collection cP of distributions over a countably infinite alphabet. Contrary to the worst case regret formulation of universal compression, finite single letter (average case) redundancy of cP does not automatically imply that the expected redundancy of describing length-n strings sampled iid from cP grows sublinearly with n. Instead, we prove that asymptotic per-symbol redundancy of universally compressing length-n iid sequences from cP is characterized by how well the tails of their single letter marginals can be universally described, and we formalize the later as the tail-redundancy of cP.