A Reinforcement-Learning Style Algorithm for Black Box Automata

The analysis of hardware and software systems is often applied to a model of a system rather than to the system itself. Obtaining a faithful model for a system may sometimes be a complex task. For learning the regular (finite automata) structure of a black box system, Angluin's $L^{*}$ algorithm and its successors employ membership and equivalence queries. The regular positive-negative inference (RPNI) family of algorithms use a less powerful capability of collecting observations for learning, with no control on selecting the inputs. We suggest and study here an alternative approach for learning, which is based on calculating utility values, obtained as a discounted sum of rewards, in the style of reinforcement learning. The utility values are used to classify the observed input prefixes into different states, and then to construct the learned automaton structure. We show cases where this classification is not enough to separate the prefixes, and subsequently remedy the situation by exploring deeper than the current prefix: checking the consistency between descendants of the current prefix that are reached with the same sequence of inputs. We show the connection of this algorithm with the RPNI algorithm and compare between these two approaches experimentally.