A Categorization of Reinforcement Learning Exploration Techniques which Facilitates Combination of Different Methods

The exploration vs. exploitation problem in reinforcement learning has received much attention lately. It is difficult to get a complete overview of all the exploration methods, and what methods can or can not be used together in a combined algorithm. We propose a categorization of exploration techniques based on the mechanism through which they generate exploratory policies. This enables plug-and-play combination of exploration techniques from each category. We show how to combine methods from different categories and demonstrate that such a combination can result in a new exploration technique which outperforms each individual method.