Co-optimization of Morphology and Behavior of Modular Robots via Hierarchical Deep Reinforcement Learning

—Modular robots hold the promise of changing their shape and even dimension to adapt to various tasks and environments. To realize this superiority, it is essential to find the appropriate morphology and its corresponding behavior simultaneously to ensure optimality of the reconfigura- tion. However, achieving co-optimization is challenging because robotic configuration and motion are interactive and coupled with each other, as well as their optimization processes. To this end, we proposed a co-optimization framework based on hierarchical Deep Reinforcement Learning (DRL), consisting of a configuration model and a motion model based on the Twin Delayed Deep Deterministic policy gradient algorithm (TD3). The two network models update asynchronously with a shared reward to ensure co-optimality. We conduct simulations and experiments with the Webots platform to validate the proposed framework, and the preliminary results show that it yields high quality optimization schemes and thus allows modular robots to be more adaptive to dynamic and multi-task scenarios.