Optimizing Multi-AAV Formation Cooperative Control Strategies With MCDDPG Approach

Abstract

In recent years, the application of autonomous aerial vehicle (AAV) devices in military, industrial, and civilian sectors has become increasingly widespread. Consequently, research on multi-AAV formation cooperative control strategies has garnered significant attention. However, current multiagent reinforcement learning algorithms often struggle with unguided exploration, making it challenging for agents to develop efficient action strategies for complex collaborative tasks. To address this issue, this article introduces a multicritic deep deterministic policy gradient (MCDDPG) algorithm. This algorithm designs a multicritic (MC) structure based on the DDPG algorithm. This structure guides AAVs using physical models for tracking and obstacle avoidance, while deep learning models are employed to facilitate cooperative coordination among AAVs. Furthermore, to address the weight allocation issue among different Critic modules in the MC structure, a dynamic difficulty priority weight optimization algorithm is implemented. This enhances the algorithm’s collaborative capabilities. To validate the collaborative planning capability of the proposed algorithm, a simulation scenario involving multicoupled tasks is designed in the multiagent particle environment (MPE). In this scenario, the MCDDPG algorithm demonstrates the fastest convergence speed and the optimal collaborative strategy, outperforming other state-of-the-art multiagent deep reinforcement learning (MADRL) algorithms currently in use.