Collaborative Collision Avoidance Approach for USVs Based on Multi-Agent Deep Reinforcement Learning

Abstract

Unmanned Surface Vehicles (USVs) have garnered extensive interest for their potential in enhancing navigation safety and efficiency. To further improve their intelligence, we propose a collaborative collision avoidance decision-making approach for USVs based on Multi-Agent Deep Reinforcement Learning (MADRL). Firstly, a collision risk assessment model is established using Closest Point of Approach and Quaternion Ship Domain to guide USVs to take timely and effective collision avoidance actions. Secondly, we adopt Deep Recurrent Q-Network algorithm to overcome the challenges posed by multi-ship scenarios, and the Decentralized Partially Observable Markov Decision Process framework is employed in it to accurately establish multi-ship collaborative collision avoidance model. Moreover, to introduce a novel collaborative collision avoidance mechanism for multiple ships, we improved the network update mechanism of DRQN: by synergistically integrating the local Q value of each agent to acquire the multi-agent joint action Q value and the network parameters are subsequently updated based on the calculated global loss. Finally, we designed various experiments in a real water to validate the applicability and efficacy of the proposed approach. The experimental results indicate that this approach has the capacity to enable the ships to make effective collision avoidance actions in accordance with navigation practices and good seamanship. Comparative analysis with Deep Q-Network (DQN), DDQN, Dueling DQN, and Artificial Potential Field underscores the superior safety and rationality of the proposed approach in collision avoidance. This research offers a new perspective for the cooperative collision avoidance decision-making and has theoretical reference significance.