Trajectory design for data collection under insufficient UAV energy: A staged actor–critic reinforcement learning approach

Abstract

Fixed-wing unmanned aerial vehicles (UAVs) offer distinct advantages for large-scale environmental sensor data collection. In forest and marine scenarios, UAVs typically depart from a fixed location, collecting data along a route, and return. Unlike existing work aiming to minimizing energy consumption on data collection task, this study focus on the scenario where a UAV’s initial energy may not be sufficient to visit all sensor nodes. We aim to maximize data collection under insufficient battery energy while make a safety return. To solve this, we adopt the twin delayed deep deterministic policy gradient (TD3) algorithm with three designed reward functions, and introduce a stage-based safe action algorithm, termed Staged Safe-Action TD3 (SS-TD3). An energy consumption model incorporating acceleration and a segmented time model are used to enhance exploration efficiency. To tackle sparse binary rewards and the suboptimal convergence of complex reward function in reinforcement learning, a staged training approach, Staged Actor–Critic based reinforcement Learning (S-ACL) is proposed, as the one of the fundamental component of SS-TD3. Experimental results show that SS-TD3 achieves the best energy efficiency compared to baselines, while S-ACL significantly improves policy performance in complex reward environments.