Dynamic resource allocation in air-ground defense via heterogeneous multi-agent tracking with cross-temporal state rewards

Abstract

The study of the target-attacker-defender (TAD) problem in three-dimensional (3D) space is of great significance to enhance the defense effectiveness in complex confrontation scenarios. However, the existing research has not fully considered the agent initiative, and the traditional reward function is difficult to motivate the defender to intercept actively; and the scenario simulation is mostly confined to the two-dimensional (2D) plane, while ignoring the differences in the distribution of the observed features, which leads to mixed information when the multi-agent algorithms deal with the non-independent and identically-distributed data. To solve these problems, this paper proposes the two-stream heterogeneous multi-agent proximal policy optimization algorithm (TSHMAPPO) algorithm. In the design of the reward mechanism, the reward relies on the combination of the previous moment’s state characteristics and the current state characteristics to provide effective reward signals for the active interception behavior of the defender, which solves the drawback of the traditional reward function that ignores the initiativ; in the observation processing, a two-stream feature extraction network is introduced into the MAPPO algorithm to realize adaptive characterization of non-independent and identically-distributed observation data, and to reduce the information confusions brought by the mixing of features. Experimental results show that the reward function design proposed in this paper improves the interception rate by 5.48% to 8.50% compared with the traditional method. The use of two-stream feature extraction network improves the defender’s interception rate by 1.47% to 10.73% in simulation experiments in different scenarios, while the TSHMAPPO algorithm has better interception performance than other comparison experiments.