A deep reinforcement learning approach and its application in multi-USV adversarial game simulation

Abstract

With the progression of unmanned surface vehicle (USV) intelligence and the maturation of cluster control technologies, intelligent decision-making methods for multi-USV adversarial games have become pivotal technological focuses. Deep reinforcement learning (DRL), a prominent subset of artificial intelligence, has recently achieved notable advancements, heralding significant potential for this field. The intrinsic curiosity module (ICM), self-play (SP), and posthumous credit assignment (POCA) are integrated with proximal policy optimization (PPO) to address the challenges of sparse reward, low sample utilization, and credit assignment in multi-USV adversarial games, and a novel proximal policy optimization algorithm (PPO-ICMSPPOCA) is finally constructed. The algorithm generates intrinsic rewards through iterative training during multi-USV adversarial games while simultaneously addressing the evaluation of each USV's specific contribution to the team and the challenge of varying numbers of USVs. A perturbation mathematical model for a USV with three degrees of freedom is established, considering the influence of external environmental disturbances and variations in the USV's state on its hydrodynamic performance in this paper. With the Unity3D and ML-Agents toolkit platforms, multi-USV adversarial game simulation scenes, which can integrate and load various reinforcement learning (RL) algorithms, have been developed. Symmetric or asymmetric game experiments of different scales are conducted in adversarial games. The experiments show that the red teams with our algorithms can learn adversarial tactics more quickly, such as troop dispersion and coordinated attacks. Over 100 episodes, the red teams with ICM, SP, and POCA achieved win rates of 88.25%, 86.75%, and 91.33%, respectively, exhibiting higher game intelligence while obtaining higher cumulative rewards.