Multi-agent Reinforcement Learning with Multi-head Attention

Multi-agent reinforcement learning(MARL) methods have become an important approach to solving the decision making problems of agents. As the environment’s complexity increases, the attention model can effectively solve the problem of information redundancy. However, the introduction of attention models in reinforcement learning may also lead to over-focusing and neglecting other potentially useful information. Moreover, the presence of attention would slow the convergence in the early stages of training. To address the above problem, we propose a divided attention reinforcement learning approach: (i) the involvement of an attention regularization term to make agents more divergent in their focus on different directions; (ii) the use of a layer normalization network structure and the use of a Pre-Layer Normalization(Pre-LN) network structure for the attention optimization in the initialization phase of training. It allows the agents to have a more stable and smooth gradient descent in the early stages of learning. Our approach has been tested in several multi-agent environment tasks. Compared to other related multi-agent methods, our method obtains higher final rewards and training efficiency.