Analysis of KNN Information Estimators for Smooth Distributions

KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence speed of this estimator for random variables with bounded support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support but also those with long tail distributions. In this paper, we analyze the convergence speed of the error of KSG estimator for smooth distributions, whose support can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive combination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions.