Historical Best Q-Networks for Deep Reinforcement Learning

The popular DQN algorithm is known to have some instability and variability which make its performance poor sometimes. In prior work, there is only one target network, the network that is updated by the latest learned Q-value estimate. In this paper, we present multiple target networks which are the extension to the Deep Q-Networks (DQN). Based on the previously learned Q-value estimate networks, we choose several networks that perform best in all previous networks as our auxiliary networks. We show that in order to solve the problem of determining which network is better, we use the score of each episode as a measure of the quality of the network. The key behind our method is that each auxiliary network has some states that it is good at handling and guides the agent to make the right choices. We apply our method to the Atari 2600 games from the OpenAI Gym. We find that DQN with auxiliary networks significantly improves the performance and the stability of games.