Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward

Robot control using reinforcement learning has become popular, but its learning process often terminates midway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentional implicit underestimation or overestimation occurs, depending on the reward design in the normal states. If the termination by failure is highly valued with the unintentional overestimation, the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in the practical use of TD learning to review the exception handling at termination. Therefore, this paper proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. This intentional underestimation is heuristically derived with the assumption of two-step transition to absorbing state. In addition, the degree of underestimation is adjusted according to the degree of steadiness at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulation results showed that the proposed method improves the success rate for 24 tasks with different reward designs from 10/24 in the conventional method to 20/24. Real-robot experiments also demonstrated that the proposed method enables to learn the optimal policy even in the case that the conventional method fails.