Fully Parameterized Dueling Mixing Distributional Q-Leaning for Multi-Agent Cooperation

Abstract

Multi-agent reinforcement learning (MARL) has been applied to many multi-agent team tasks, such as multi-robot swarm control. Distributional Value Function Factorization (DFAC) follows the Distributional-Individual-Global-Max (DIGM) principle, which forces the individual's optimal action to be in accordance with the optimal joint action at all times. However, this principle leads to grade inflation, which limits agents to exploring better strategies than before. We focus on this issue and propose a novel MARL method named dueling mixing distributional Q-learning with fully parameters (FDMIX). Firstly, a parametric individual value network generates an individual distribution function and a utility value function, while the fractions are obtained through a fraction proposal network. Secondly, the conversion mixing network obeys a new advantage-based DIGM principle to generate a joint distribution action value based on the global state. Finally, we incorporate an N-step return-based loss function to achieve stable and efficient training. Our extensive tests on the multiple-particle environment and StarCraft II show that our method performs better than state-of-the-art algorithms noticeably.