Formation control and intention compensating of AUVs using multi-agent reinforcement learning and predict network

Abstract

Autonomous Underwater Vehicles (AUVs) have played an important role in numerous marine tasks, such as resource exploration, hydrological data acquisition, rescue operations, and military missions. In contrast to single AUV deployment, multi-AUV formations exhibit higher efficiency and improved task completion rates. Recently, multi-agent reinforcement learning (MARL) has emerged as a promising technique for AUV formation control. Nevertheless, conventional MARL approaches often suffer from instability in formation shapes, especially when managing a large number of AUVs. Additionally, communication delay and information dropout can further compromise formation performance. In this paper, we propose a novel method called Policy Compensate Multi-agent Twin Delayed Deep Deterministic Policy Gradient (PC-MATD3), which integrates imitation learning (IL) with MARL to improve formation stability. The proposed framework is designed to alleviate adverse effects caused by communication interruptions or information delays. We define distance and angular errors as key performance metrics and evaluate our method through two distinct simulation scenarios. Experimental results show that, under ideal communication conditions, our approach substantially reduces formation errors and improves overall stability. Additionally, in scenarios involving communication dropouts, the proposed method effectively predicts the positions of neighboring AUVs, enabling the restoration of the desired formation geometry.