Beamforming design and trajectory optimization for integrated sensing and communication supported by multiple UAVs based on DRL

Abstract

With the rapid development of Unmanned aerial vehicle (UAV) technology and the high flexibility and maneuverability of UAV itself, UAV will play a very important role in the development of integrated sensing and communication (ISAC) in the future. In this paper, the communication and sensing system supported by multiple UAVs is studied. And we propose a new ISAC balance mode (BISAC). In this mode, the sensing time is set reasonably according to the number of potential targets (PTs) and sensing requirements while the UAV is communicating with ground equipment (GEs), so as to reduce the interaction between communication and sensing and improve the utilization of resources. We also introduce the Age of Information (AoI) to measure the freshness of GEs' data information in order to reduce the delay. Therefore, our goal is to minimize the Average AoI of GEs by jointly optimizing UAV trajectory, user association, target sensing selection and communication and sensing beamforming while maintaining communication quality and sensing requirements. In order to obtain long-term AoI performance and effectively solve non-convex problems with continuous and discrete variables, we propose a deep reinforcement learning (DRL) algorithm based on a combination of deep deterministic policy gradient (DDPG) and Dueling Double Deep Q networks (D3QN). Continuous and discrete variables in the system are processed by invoking a DDPG and D3QN. Specifically, we have improved DDPG's actor-critic structure by incorporating D3QN, which utilizes the actor portion of DDPG to search for optimal communication and sensing beams. At the same time, the critic part of DDPG is combined with D3QN to select the optimal flight direction of UAV. Simulation results show that the proposed DDPG-D3QN algorithm has better stability, faster convergence rate, and higher reward than existing DRL-based methods.