Unsupervised Discovery of Transitional Skills for Deep Reinforcement Learning

Abstract

By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering using multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose a novel unsupervised reinforcement learning approach to learn transitional skills in addition to pursuing diverse primitive skills. By introducing an extra latent variable for exploring the dependence between skills, our method discovers both primitive and transitional skills by optimizing a novel information theoretic objective. Considering various robotic tasks, our results demonstrate the effectiveness on learning both diverse primitive skills and transitional skills, and further exhibit the superiority of our method in smooth transition of skills over the baselines. Videos of transitional skills can be found on the project website: https://sites.google.com/view/udts-skill.