Vision-based attention deep q-network with prior-based knowledge

Abstract

Vision-based reinforcement learning (RL) is a potent algorithm for addressing tasks related to visual behavioural decision-making; nevertheless, it operates as a black-box, directly training models with images as input in the end-to-end fashion. Therefore, to elucidate the underlying mechanisms of the model and the agent’s focus on different features during the decision-making process, a vision-based attention (VA) mechanism is introduced into vision-based RL in this paper. A prior-based mechanism is introduced to address the issue of instability in the attention maps observed by the agent when attention mechanisms are directly integrated into network updates that results in an increase in single-step errors and larger cumulative errors. Thus, a vision-based attention deep Q-network (VADQN) method with a prior-based mechanism is proposed. Specifically, prior attention maps are obtained using a learnable Gaussian filtering and a spectral residual method. Next, the attention maps are fine-tuned using a self-attention (SA) mechanism to enhance their performance. During training, both the attention maps and the parameters of the policy network are concurrently trained to ensure explanations of the regions of interest during online training. Finally, a series of ablation experiments are conducted on Atari games to compare the proposed method with humans, convolutional neural networks, and other approaches. The results demonstrate that the proposed method not only reveals the regions of interest attended to by DRL during the decision-making process but also enhances DRL performance in certain scenarios. This approach provides valuable insights for understanding and improving the performance of DRL in visual decision-making tasks.