TUCA-HER: An Improved HER for Robot Manipulation Skill Learning via Trajectory Utility and Conservative Advantage

Abstract

In the realm of multi-goal reinforcement learning for robot manipulation, effectively addressing sparse rewards has been a key challenge. The hindsight experience replay (HER) mechanism has provided notable advancements in this domain, yet its efficiency and adaptability still require further improvement. This paper introduces TUCA-HER for robot manipulation skill learning via Trajectory Utility and Conservative Advantage. We start by computing trajectory utility for experience samples collected in the early stages of training, which allows for dynamic relabeling and significantly enhances sample efficiency. Furthermore, we integrate conservative advantage learning into the actor-critic framework, reshaping rewards to construct TUCA-HER. Finally, we apply TUCA-HER to robot manipulation skill learning tasks, providing details on algorithmic implementation and complexity analysis. Evaluations conducted on OpenAI Fetch and Hand environments demonstrate TUCA-HER's superior performance in sample efficiency and task success rate compared to other algorithms. Notably, in the FetchPickAndPlace task, TUCA-HER showcases a remarkable 46% improvement over the Double experience replay buffer Adaptive Soft Hindsight Experience Replay (DAS-HER). Furthermore, Sim-to-Real experiments are conducted to validate the effectiveness of TUCA-HER in real-world environments.