Research on Dueling Double Deep Q Network Algorithm Based on Single-Step Momentum Update

Abstract

Vehicle behavior decision control plays a crucial role in the development of autonomous driving. However, existing autonomous driving behavior decision control algorithms based on deep reinforcement learning face several challenges, such as low efficiency in updating target network data and a lack of effective balancing between old and new experiences. To address these issues, this paper proposes a dueling double deep Q network (dueling DDQN) algorithm based on a single-step momentum update mechanism. Firstly, a single-step momentum update mechanism is designed to significantly improve the update speed of target network parameters and achieve a balanced weighting of old and new experiences during the parameter update process. Subsequently, the network structures of dueling networks and DDQNs are integrated to enhance the understanding capability of autonomous vehicles concerning their current states. Finally, tests are conducted on the OpenAI Gym simulation platform to validate the effectiveness of the proposed algorithm. The results verified that the dueling DDQN algorithm with single-step momentum updates contributes to improving the convergence speed of autonomous driving car behavior decisions. Compared with the DQN and DDQN algorithms, the proposed algorithm achieved a success rate increase of 6.0 and 8.4 percentage points in the challenging three-lane highway Test Scenario 1, and a success rate increase of 16.7 and 2.9 percentage points in Test Scenario 2, respectively. These findings demonstrate a safer and more efficient performance in autonomous driving decision-making.