SMART: Sequential Multiagent Reinforcement Learning With Role Assignment Using Transformer

Multiagent reinforcement learning (MARL) has received increasing attention and been used to solve cooperative multiagent decision-making and learning control tasks. However, the high complexity of the joint action space and the nonstationary learning process are two major problems that negatively impact on the sample efficiency and solution quality of MARL. To this end, this article proposes a novel approach named sequential MARL with role assignment using transformer (SMART). By learning the effects of different actions on state transitions and rewards, SMART realizes the action abstraction of the original action space and the adaptive role cognitive modeling of multiagent, which reduces the complexity of the multiagent exploration and learning process. Meanwhile, SMART uses causal transformer networks to update role assignment policy and action selection policy sequentially, alleviating the influence of nonstationary multiagent policy learning. The convergence characteristic of SMART is theoretically analyzed. Extensive experiments on the challenging Google football and StarCraft multiagent challenge are conducted, demonstrating that compared with mainstream MARL algorithms such as MAT and HAPPO, SMART achieves a new state-of-the-art performance. Meanwhile, the learned policies through SMART have good generalization ability when the number of agents changes.