A Deep Reinforcement Learning Based Energy Management Strategy for Fuel-Cell Electric UAV

Abstract

Electric propulsion UAV powered by hybrid power system consisting of fuel cells and lithium batteries have attracted significant attention for long endurance and zero emission. Different dynamic characteristics for variable power load demanding which can be stochastically affected by the UAV’s flight air dynamic disturbance are difficult to be modeled with energy management system (EMS). In this paper, a Deep Reinforcement Learning (DRL) algorithm, namely twin-delayed Deep Deterministic policy gradient (TD3), is adopted to derivate EMS for hybrid electric UAV which can avoid performance degradation from uncertainty of power system model and curse of dimensionality of traditional algorithm. The simulation results indicate that the TD3-based DRL strategy is able to coordinate multiple electric power sources based on their natural power characteristics, satisfy different flight profiles of UAV. Furthermore, the performances of TD3, Deep Q-Networks (DQN), Deep Deterministic policy gradient (DDPG) and Dynamic Programming (DP) algorithms with different parameters in EMS of hybrid electric UAV were compared and the effectiveness of the algorithm was verified by digital simulation. Comparative results also illustrate that the proposed TD3 method outperforms other two methods in solving multi-objective optimization energy management problem, in terms of hydrogen consumptions, system efficiency and battery’s state of charge (SOC) sustainability.