Enhancing multi-agent reinforcement learning via world model assisted single-agent population policies in multi-UAV cooperative-competitive scenario

Abstract

Multi-agent reinforcement learning (MARL) with limited search is prone to becoming trapped in local optima and struggles to adapt to opponents’ changing intentions and strategies in cooperative-competitive environments. Evolutionary strategies (ES) are a promising alternative that have been applied to RL owing to their diverse exploration characteristics to form a hybrid framework. However, most such methods require population policies to interact within the environment for Monte Carlo (MC) evaluation or to filter experience using Q-functions, leading to low sample efficiency. Additionally, existing methods struggle to maintain policy quality and avoid catastrophic forgetting while exploring. Our objective is to enhance MARL in mixed environments through single-agent population policies to improve the sample efficiency and mitigate the aforementioned issues with ES. To achieve this, we propose a world-model-assisted cross entropy method (CEM)-MARL approach. This world model enables inferring opponents’ mental states and predicting and evaluating future trajectories. CEM is used to update population policies and retain the top K/2 old policies. Subsequently, predicted future experiences are reused for updating the MARL policy and evaluated to perform quality-assured mutations on the top K/2 population policies. Furthermore, the elite policy guides the MARL, and the MARL policy guides the population. Experimental results in multiple unmanned aerial vehicle (multi-UAV) game scenarios show that our method accelerates learning by 40 times compared to model-free MARL and nearly 10 times compared to our version without ES. This increases the learning efficiency and enhances the performance of the hybrid framework that incorporates collaborative learning and evolutionary processes.