Exception-based reinforcement learning

In this paper we develop a method using temporally abstract actions to solve Markov decision processes. The basic idea of our method is to define some kind of procedures to control the agent's behavior. These procedures contain a rule constraining actions the agent has to choose. This rule is applied except if some conditions (which we call exceptions) are fulfilled. In this case we relax constraints on actions. We develop a way to propagate states that have created an exception to a rule, to help the agent to escape from blocked situations or locally optimal solutions. We illustrate the method using the "Sokoban" game. We compare the method empirically with flat Q-learning. On the proposed tests, learning time is drastically reduced as is the memory required to save the Q-values.