An effective exploration method based on N-step updated Dirichlet distribution and Dempster–Shafer theory for deep reinforcement learning

Abstract

Deep reinforcement learning (DRL) has been regarded as a promising approach for solving decision-making problems. However, how to enhance the agent exploration ability is still an extremely challenging issue for existing methods, especially under sparse rewards. Facing with this challenge, we propose a novel efficient exploration method, which can comprehensively consider the uncertainty of the environment and the uncertainty of Q function, so as to improve the agent exploration efficiency. Specifically, we first construct an exploration policy by n-step updated Dirichlet distribution to implement the adaptive exploration of the agent to the environment, which can reduce the uncertainty of the agent about the environment to achieve global efficient exploration. Next, a state–action basic probability assignment (BPA) is constructed based on the Dempster–Shafer theory. On this basis, an interval Q function is designed by combining BPA and belief interval, which can effectively characterize the uncertainty of the Q function to achieve deep exploration. Then, the proposed method is applied to classic DRL algorithms, deep Q-network (DQN) and double DQN (DDQN), two novel algorithms are proposed. Finally, under a series of sparse external reward tasks, experimental results show that our proposed algorithms outperform several state-of-the-art DRL algorithms in term of exploring efficiency.