Uniform Estimates on Length of Programs and Computing Algorithmic Complexities for Quantitative Information Measures

Abstract

Shannon entropy and Kolmogorov complexity are two conceptually distinct information metrics since the latter is based on probability distributions while the former is based on program size. All recursive probability distributions, however, are known to have an expected Up to a constant that solely depends on the distribution, the Kolmogorov complexity value is equal to its Shannon entropy. We investigate if a comparable correlation exists between Renyi and Havrda- Charvat Entropy entropies order α, indicating that it is consistent solely with Renyi and Havrda- Charvat entropies of order 1. Kolmogorov noted that the characteristics of Shannon entropy and algorithmic complexity are comparable. We examine a single facet of this resemblance. Specifically, linear inequalities that hold true for Shannon entropy and for Kolmogorov complexity. As it happens, the following are true: (1) all linear inequalities that hold true for Shannon entropy and vice versa for Kolmogorov complexity; (2) all linear inequalities that hold true for ranks of finite subsets of linear spaces for Shannon entropy; and (3) the reverse is untrue.