Estimation of Statistical Manifold Properties of Natural Sequences using Information Topology

Abstract

Modeling unknown natural sequences is a challenging area. Here we consider an information theoretic approach for analyzing probabilistic natural sequences in the context of synthetic languages, which are characterized by having no available language models. Based on the notion of efficient short-term entropy estimators, we examine the concept of extending information geometry to information topology as a method of characterizing natural sequences. A normalized relative difference entropy method is described, which is required to apply the technique to sub-word models derived from natural sequences. Visualization of information topological spaces is considered, and some aspects are considered for future work. The approach is shown to provide potential as a new method for modeling the probabilistic structure of synthetic language sequences.