An information theoretic proof of the Chernoff-Hoeffding inequality

Abstract

The Chernoff bound is a well-known upper bound on the tail of binomial distributions of parameter 1/2 involving the binary entropy function. Hoeffding's inequality (or the Chernoff-Hoeffding inequality) is a generalization for binomial distributions of parameter $1-1/q$, involving the q-ary entropy function (with $q\ge 2$), which can be written in terms of the Kullback-Leibler divergence and is related to the bound in Fano's inequality. We give an information theoretic proof of that bound, and sketch some applications to channel and source coding. We also derive a refined bound which is always sharper.