Addressing Different Goal Selection Strategies In Hindsight Experience Replay With Actor-Critic Methods For Robotic Hand Manipulation

Abstract

One of the most challenging problems in reinforcement learning is dealing with minimal rewards obtained from an environment. We present a combined technique of Twin Delayed Deep Deterministic Policy Gradient known as TD3, an off-policy Reinforcement Learning algorithm with Hindsight Experience Replay (HER). This combined technique allows for sampleefficient learning from sparse and binary rewards and avoids the need for complicated reward engineering. We use the challenge of moving things with a robotic arm to illustrate our methodology. We specifically tested six different tasks: pushing, sliding, picking up and placing in the Fetch environment, as well as manipulating a block, an egg, or a pen with our hands. We solely use binary rewards every time to indicate whether or not a task has been performed. In a comparative study, we primarily concentrate on the impact of various goal selection strategies of HER replay butter on both DDPG and TD3. We discovered that HER was crucial in enabling training in these demanding situations.