Mixed-Reward Multiagent Proximal Policy Optimization Method for Two-on-Two Beyond-Visual-Range Air Combat

Abstract

With recent advances in airborne weapons, modern air combats tend to be accomplished in the beyond-visual-range (BVR) phase. Multiaircraft cooperation is also required to adapt to the complexities of modern air combats. The scale of the traditional rule-based expert system will become incredible in this case. In view of this, a mixed-reward multiagent proximal policy optimization (MRMAPPO) method is proposed in this article that is used to help train cooperative BVR air combat tactics via adversarial self-play. First, a two-on-two BVR air combat simulation platform is established, and the combat game is modeled as a Markov game. Second, centralized training with decentralized execution architecture is established. Multiple actors are involved in the architecture, each corresponding to a policy that generates a specified kind of command, e.g., the maneuvering and firing command. Moreover, in order to accelerate training as well as enhance the stability of the training process, four optimization mechanisms are introduced. The experimental section discusses how the effectiveness of the MRMAPPO is verified with comparative and ablation experiments, along with several air combat tactics that emerge in the training process.