Fast UAV path planning in urban environments based on three-step experience buffer sampling DDPG

The path planning of Unmanned Aerial Vehicle (UAV) is a critical issue in emergency communication and rescue operations, especially in adversarial urban environments. Due to the continuity of the flying space, complex building obstacles, and the aircraft's high dynamics, traditional algorithms cannot find the optimal collision-free flying path between the UAV station and the destination. Accordingly, in this paper, we study the fast UAV path planning problem in a 3D urban environment from a source point to a target point and propose a Three-Step Experience Buffer Deep Deterministic Policy Gradient (TSEB-DDPG) algorithm. We first build the 3D model of a complex urban environment with buildings and project the 3D building surface into many 2D geometric shapes. After transformation, we propose the Hierarchical Learning Particle Swarm Optimization (HL-PSO) to obtain the empirical path. Then, to ensure the accuracy of the obtained paths, the empirical path, the collision information and fast transition information are stored in the three experience buffers of the TSEB-DDPG algorithm as dynamic guidance information. The sampling ratio of each buffer is dynamically adapted to the training stages. Moreover, we designed a reward mechanism to improve the convergence speed of the DDPG algorithm for UAV path planning. The proposed TSEB-DDPG algorithm has also been compared to three widely used competitors experimentally, and the results show that the TSEB-DDPG algorithm can archive the fastest convergence speed and the highest accuracy. We also conduct experiments in real scenarios and compare the real path planning obtained by the HL-PSO algorithm, DDPG algorithm, and TSEB-DDPG algorithm. The results show that the TSEB-DDPG algorithm can archive almost the best in terms of accuracy, the average time of actual path planning, and the success rate.