Inferring the structure of a Markov Chain from its output

Abstract

To make matters simpler, assume an upper bound n is known on the number of states in the Markov Chain. Fortunately, there are only a finite number of finite state machines with at most n states. For each such finite state machine, one can hypothesize that it underlies the Markov Chain. Then for any finite string of outputs from the Markov Chain, one can estimate probabilities on the transitions. These probabilities are asymptotically equal to the true probabilities (if the underlying finite state machine is the correct one) with probability 1. We attach these probabilities to the transitions and from these determine the entropy of each n-state machine. We show that with probability 1, any machine that has maximum entropy (in the limit as the length of the output string goes to infinity) is a correct guess.