MaxEnt Dreamer: Maximum Entropy Reinforcement Learning with World Model

Abstract

Model-based reinforcement learning algorithms can alleviate the low sample efficiency problem compared with modelfree methods for control tasks. However, the learned policy's performance often lags behind the best model-free algorithms since its weak exploration ability. Existing model-based reinforcement learning algorithms learn policy by interacting with the learned world model and then use the learned policy to guide a new round of world model learning. Due to weak policy exploration ability, the learned world model has a large bias. As a result, it fails to learn the globally optimal policy on such a world model. This paper improves the learned world model by maximizing both the reward and the corresponding policy entropy in the framework of maximum entropy reinforcement learning. The effectiveness of applying the maximum entropy approach to model-based reinforcement learning is supported by the better performance of our algorithm on several complex mujoco and deepmind control suite tasks.