Bayesian curriculum generation in sparse reward reinforcement learning environments

Abstract

This paper introduces the Bayesian Curriculum Generation Algorithm, a sophisticated approach for curriculum learning in sparse reward reinforcement learning contexts. Diverging from traditional methodologies, this algorithm utilizes Bayesian networks to dynamically create tasks by altering problem parameters, thereby impacting task difficulty. It operates independently from the core reinforcement learning algorithm, enabling compatibility with a variety of RL techniques. A notable feature of our algorithm is its capability for unsupervised task classification, utilizing a clustering process applicable to both image outputs and scalar values. This method efficiently categorizes tasks based on difficulty, circumventing the need for exhaustive training for each task. However, the effectiveness of this approach is contingent upon the presence of definable parameters within the environment and necessitates domain expertise to determine the appropriate tool, be it image output or scalar parameter analysis. The algorithm selects tasks from a curated pool corresponding to specific difficulty levels and adapts according to the agent’s performance. Successful task completion triggers the generation of more complex tasks, whereas encountering challenges results in the maintenance or minor adjustment of task complexity. This adaptive feature significantly enhances the efficiency of the learning process. Empirical evaluations conducted in various environments, characterized by maze-like structures, discrete or continuous settings, and the presence of adversarial entities hindering the agent’s mission, demonstrate the algorithm’s efficacy and its superiority over conventional methods. The Bayesian Curriculum Generation Algorithm represents a significant advancement in reinforcement learning, providing a dynamic and adaptable solution for complex learning challenges.