A learning-based algorithm for turn-based orbital pursuit-evasion problem with reaction-time delay

Abstract

In this paper, we propose an artificial intelligence-based methodology to investigate the impulsive orbital pursuit-evasion problem, taking into account the often-neglected factor of reaction time delay. This study defines this scenario as a Delayed Turn-based Orbital Pursuit-Evasion Game (DT-OPEG) and establishes relevant concepts and definitions based on orbital dynamics and game theory. Subsequently, considering several constraints inherent in real-world missions, including nonlinear orbital dynamics, maneuvering capabilities, fuel reserves, and mission duration, we formulate the problem modeling for DT-OPEG. To address the complexity of this problem, especially the challenge of incorporating action delays, we propose a Delayed Turn-based Multi-Agent Deep Deterministic Policy Gradient (DT-MADDPG) algorithm. The establishment process of this algorithm includes establishing a Turn-based Markov Decision Process (T-MDP) model with reaction-time delay, constructing a turn-based training framework, developing a network architecture based on MADDPG, and designing reward functions. Finally, simulation analyses are conducted for both two-dimensional and three-dimensional DT-OPEG scenarios, confirming the effectiveness of the proposed algorithm and demonstrating the winning mechanism in this type of game.