Robust walking motion generation for biped robots using manipulability-based reinforcement learning

Abstract

In reinforcement learning, designing an effective reward function is essential for developing and controlling humanoid robots. The criteria for replicating human learning and achieving human-like responses in bipedal robots remain unclear. Integrating kinematic and dynamic characteristics into the reward function, along with the use of detailed models, can enhance efficiency and robustness. This study proposes a novel manipulability-based reward function within an end-to-end learning framework, enabling the agent to autonomously generate robust, real-time movements. Incorporating the kinematic manipulability index into the proposed reward function significantly improves the robot's locomotion behavior and ability to handle disturbances. Results indicate that incorporating kinematic manipulability into training enhances the robot's forward speed and improves its ability to handle sagittal and lateral disturbances, as well as uncertainties in length and weight distribution. Furthermore, compared to a classical hierarchical controller, the trained agent attained higher speeds and demonstrated superior disturbance handling, validating the effectiveness of the proposed learning-based approach. These findings highlight the significance of incorporating kinematic manipulability into the reward function to enhance the agility and adaptability of bipedal robots.