An Ensemble MARL Approach for Heterogeneous UAV Swarm Target Search in 3D Space

Abstract

In recent years, unmanned aerial vehicles (UAVs) have become increasingly pivotal in cooperative searches.However, the performance of using a homogeneous UAV swarm for search is often limited because a single class of UAVs cannot guarantee accuracy and coverage simultaneously. This work tackles the challenge of target search in a large-scale three-dimensional (3D) environment using a heterogeneous UAV swarm. Specifically, we design a novel search scheme that leverages the complementary strengths of two UAV types: fixed-wing UAVs, which enable swift area coverage at high altitudes, and multi-rotor UAVs, which provide high-precision detection at low altitudes. To address the stochastic optimization problem posed by unknown targets, we develop a multi-agent reinforcement learning (MARL) scheme—where multiple agents learn to optimize their strategies through interactions with each other and the environment—for efficient target search. To ensure the safety of UAVs, an action masking scheme is integrated into the MARL framework to avoid collisions. Moreover, to further enhance decision robustness and search efficiency, we propose an Ensemble MARL framework that aggregates multiple independently trained networks via a majority voting mechanism, and further present our Ensemble QMIX (E-QMIX) algorithm. Furthermore, we provide a mathematical analysis showing that E-QMIX increases the probability that each agent selects its optimal action, thereby strengthening the robustness of joint decision-making. Simulation results demonstrate that our E-QMIX outperforms classic single-network MARL benchmarks in both search efficiency and coverage rate.