On the discontinuity of the Shannon information measures and typical sequences

Abstract

It is well known that the Shannon information measures are continuous functions of the probability distribution when the support is finite. This, however, does not hold when the support is countably infinite. In this paper, we investigate the continuity of the Shannon information measures for countably infinite support. With respect to a distance based on the Kullback-Liebler divergence, we use two different approaches to show that all the Shannon information measures are in fact discontinuous at all probability distributions with countably infinite support