Communication Complexity of Exact Sampling Under Rényi Information

Abstract

We study the problem of exact sampling under an exponential communication cost, specifically Campbell’s average codeword length $\boldsymbol {L(t)}$ of order $\boldsymbol {t}$ , and Rényi’s entropy. We provide a lower bound on the Campbell cost of exact sampling that grows approximately as $\boldsymbol {D_{1/\alpha }(P||Q)}$ , the Rényi divergence of order $\boldsymbol {1/\alpha }$ , with $\boldsymbol {\alpha = 1 / 1 + t}$ . Using the Poisson functional representation of Li and El Gamal (2017), we prove an upper bound on $\boldsymbol {L(t)}$ whose leading Rényi divergence term has order within $\boldsymbol {\epsilon }$ of that of the lower bound. Our results reduce to the bounds of Harsha et al. (2010) as $\boldsymbol {\alpha \to 1}$ . We also provide numerical examples comparing the bounds in the cases of normal and Laplacian distributions, demonstrating that the upper and lower bounds are typically within 5-10 bits of each other. Our results characterize exactly the optimal asymptotic Campbell cost $\boldsymbol {L(t)}$ per sample as the number of independent and identically distributed (i.i.d.) samples grows to infinity. We show that under the exponential cost, any causal sampler performs strictly worse asymptotically than noncausal samplers. This contrasts with the case of expected message length, where both causal and noncausal samplers have the same optimal asymptotic cost.