Improved proximal policy optimization for UAV tracking in complex environments

Abstract

Unmanned Aerial Vehicles (UAVs) operating in urban environments face critical challenges in dynamic field of view (FOV) management and obstacle avoidance. To address these issues, this paper proposes an improved Proximal Policy Optimization algorithm (I-PPO) that integrates seven key enhancements, including reward scaling, gradient clip, and others. This algorithm improves sample efficiency and reduces policy oscillation in complex environments, in which we have developed a three-dimensional simulation environment capable of multi-terrain parametric modeling that integrates weather-related FOV attenuation models and intelligent dynamic obstacle modules. Focusing on the tracking task, the study designs a reward function based on a hierarchical penalty system and priority rules. This approach ensures operational safety while maximizing target vehicle visibility, thereby optimizing agent performance under environmental uncertainties. Experimental results demonstrate that in plain environments, I-PPO yields a 2.9-fold increase in mean cumulative reward and extends target tracking duration by a factor of 2.7 compared to the standard PPO. In hilly terrain, I-PPO maintains reward performance comparable to its plain environment baseline, exhibiting merely a 2% performance degradation, confirming terrain adaptability. In mountainous terrain, while it shows a 12% reward reduction versus hilly terrain, it exhibits a 38.9% reduction in reward variance (measured by IQR) compared to Discrete Soft Actor–Critic (DSAC), this demonstrates significant robustness enhancement. In scenarios with 10 intelligent dynamic obstacles, the algorithm achieves stable convergence within 984 time units and demonstrates equivalent robustness under weather-induced FOV attenuation across multi-terrain environments. Furthermore, Theoretical analysis confirms the method’s compliance with policy gradient convergence requirements.