Multi-Dueling framework for multi-agent reinforcement learning

Abstract

In real-world tasks, multiple agents often need to coordinate with one another due to their individual private observations and restricted communication abilities. A representative research direction is the deep multi-agent reinforcement learning value decomposition, which decomposes the global shared joint action value function $Q_{tot}(\tau ,u)$ into their respective action value functions $Q_i({\tau }_i,u_i)$ to guide the behavior of individuals. They all follow the IGM (Individual-Global-Max) principle, obeying the addable assumption and the monotonic assumption to support effective local decision making. However, to achieve scalability, existing MARL algorithms often compromise either the expressive power of their value function representations or the consistency of the IGM principles. This compromise can potentially result in instability or poor performance when dealing with complex tasks. In this paper, we introduce a novel algorithm called MDF—a Multi-Dueling Framework for Multi-Agent Reinforcement Learning. We innovatively propose the V-IGM constraint principle and correct the incomplete expression of the constant term $c\left(\tau \right)$ of the Qatten algorithm to further refine the decomposition of the joint action value function $Q_{tot}(\tau ,u)$. The MDF algorithm innovatively utilizes the Dueling Network architecture for decomposing the joint action value function $Q_{tot}(\tau ,u)$. Additionally, it incorporates the multi-attention mechanism to achieve an even more refined decomposition of $Q_{tot}(\tau ,u)$. Experiments show that MDF algorithm outperforms the most advanced MARL algorithm in StarCraft$\Pi$ maps (e.g. 3 m, 8 m, 2m-vs-1z, 2m-vs-1sc, 2s3z, 3s-vs-4z, 3s-vs-5z, 3s5z, 1c3s5z, bane-vs-bane).