Combat game strategy for unmanned surface vessels based on modified deep reinforcement learning

Abstract

Unmanned Surface Vessel (USV) are transitioning from auxiliary roles to primary combat platforms, emerging as a pivotal element in advancing the technological capabilities of maritime unmanned combat systems. Artificial intelligence technologies, such as reinforcement learning (RL), to equip USV with autonomous combat decision-making capabilities is an important research direction. To address the issue of sparse rewards in reinforcement learning, this paper proposes an improved Proximal Policy Optimization algorithm based on situational field (SF-PPO) for training USV combat strategies. Firstly, a combat game environment involving two USV is constructed, with decision-making models for both sides developed using RL-based and rule-based approaches, respectively. Subsequently, a situational field model for USV combat is designed based on the operational mechanics of USVs, and the reward function of PPO is improved using the situational field model. Finally, the SF-PPO model for USV is trained in randomized combat scenarios. Meanwhile, a series of multi-scenario tests involving dual-USVs combat game are designed to evaluate the model’s performance. The results demonstrate that the proposed SF-PPO model exhibits superior generalization capabilities and convergence efficiency. It achieves lower loss rates compared to other decision-making models and is capable of executing complex tactical decisions across various scenarios, showcasing a higher level of artificial intelligence.