Double mixing networks based monotonic value function decomposition algorithm for swarm intelligence in UAVs

Abstract

In multi-agent systems, particularly when facing challenges of partial observability, reinforcement learning demonstrates significant autonomous decision-making capabilities. Aiming at addressing resource allocation and collaboration issues in drone swarms operating in dynamic and unknown environments, we propose a novel deep reinforcement learning algorithm, DQMIX. We employ a framework of centralized training with decentralized execution and incorporate a partially observable Markov game model to describe the complex game environment of drone swarms. The core innovation of the DQMIX algorithm lies in its dual-mixing network structure and soft-switching mechanism. Two independent mixing networks handle local Q-values and synthesize them into a global Q-value. This structure enhances decision accuracy and system adaptability under different scenarios and data conditions. The soft-switching module allows the system to transition smoothly between the two networks, selecting the output of the network with smaller TD-errors to enhance decision stability and coherence. Simultaneously, we introduce Hindsight Experience Replay to learn from failed experiences. Experimental results using JSBSim demonstrate that DQMIX provides an effective solution for drone swarm game problems, especially in resource allocation and adversarial environments.