Developmentally Synthesizing Earthworm-Like Locomotion Gaits with Bayesian-Augmented Deep Deterministic Policy Gradients (DDPG)

Abstract

In this paper, a reinforcement learning method is presented to generate earthworm-like gaits for a hyperredundant earthworm-like manipulator robot. Partially inspired by human brain’s learning mechanism, the proposed learning framework builds its preliminary belief by first starting with adapting rudimentary gaits governed by a generic kinematic knowledge of undulatory, sidewinding and circular patterns. The preliminary belief is then represented as a prior ensemble to learn new gaits by leveraging apriori knowledge and learning a policy by inferring posterior over prior distribution. While the fundamental idea of incorporating Bayesian learning with reinforcement learning is not new, this paper extends Bayesian actor-critic approach by introducing augmented prior-based directed bias in policy search, aiding in faster parameter learning and reduced sampling requirements. We show results on an in-house built 10-DoF earthworm-like robot that exhibits adaptive development, qualitatively learning different locomotion modes, while given with only rudimentary generic gait behaviors. The results are compared against deterministic policy gradient method (DDPG) for continuous control as the baseline. We show that our proposed method can characterize effective performance over DDPG, and it also achieves faster kinematic indexes in various gaits.