Developmental Reinforcement Learning through Sensorimotor Space Enlargement

Abstract

In the framework of model-free deep reinforcement learning with continuous sensorimotor space, we propose a new type of transfer learning, inspired by the child development, where the sensorimotor space of an agent grows while it is learning a policy. To decide how the dimensions grow in our neural network based actor-critic, we add new developmental layers to the neural networks which progressively uncover some dimensions of the sensorimotor space following an Intrinsic Motivation heuristic. To mitigate the catastrophic forgetting problem, we take inspiration from the Elastic Weight Constraint to regulate the learning of the neural controller. We validate our approach using two state-of-the-art algorithms (DDPG and NFAC) on two high-dimensional environment benchmarks (Half-Cheetah and Humanoid). We show that searching first for a suboptimal solution in a subset of the parameter space, and then in the full space, is helpful to bootstrap learning algorithms, and thus reach better performances in fewer episodes.