A novel trajectories optimizing method for dynamic soaring based on deep reinforcement learning

Abstract

Dynamic soaring, inspired by the wind-riding flight of birds such as albatrosses, is a biomimetic technique which leverages wind fields to enhance the endurance of unmanned aerial vehicles (UAVs). Achieving a precise soaring trajectory is crucial for maximizing energy efficiency during flight. Existing nonlinear programming methods are heavily dependent on the choice of initial values which is hard to determine. Therefore, this paper introduces a deep reinforcement learning method based on a differentially flat model for dynamic soaring trajectory planning and optimization. Initially, the gliding trajectory is parameterized using Fourier basis functions, achieving a flexible trajectory representation with a minimal number of hyperparameters. Subsequently, the trajectory optimization problem is formulated as a dynamic interactive process of Markov decision-making. The hyperparameters of the trajectory are optimized using the Proximal Policy Optimization (PPO2) algorithm from deep reinforcement learning (DRL), reducing the strong reliance on initial value settings in the optimization process. Finally, a comparison between the proposed method and the nonlinear programming method reveals that the trajectory generated by the proposed approach is smoother while meeting the same performance requirements. Specifically, the proposed method achieves a 34% reduction in maximum thrust, a 39.4% decrease in maximum thrust difference, and a 33% reduction in maximum airspeed difference.