An internal supplemental action measurement to increase the gap of action values and reduce the sensitivity to overestimation error

Abstract

ABSTRACT Reinforcement learning holds considerable promise to help address sequential decision-making problems, in which Q-learning is one of the most used algorithms. However, Q-learning suffers from overestimation errors, especially when action values in the same state are similar. To reduce such damages, we introduce an internal supplemental action measurement based on the variation of the expected state values between a state transition, which can measure the action causing the state transition. For the reason that the internal supplemental action measurement can increase or decrease the action values according to the action performance, it can increase the gap of the action values, thus reducing the sensitivity to the overestimation error. The experimental results in the Markov chain and the video games demonstrate the performance advantage of applying the internal supplemental action measurement, in which the mean evaluating scores with the internal supplemental action measurement are 131.6% in SpaceInvaders, 187.9% in Seaquest, and 176.6% in Asterix respectively of that without the internal supplemental action measurement.