Distributed scalable multi-agent reinforcement learning with intrinsic-episodic dual exploration

Abstract

Cooperative multi-agent reinforcement learning still faces challenges in multi-agent exploration and data-efficiency. In this paper, we propose a practical framework named Distributed Scalable Multi-Agent Reinforcement Learning with Intrinsic-Episodic Dual Exploration (SIEMA) to tackle these challenges. Under the widely-applied assumption of centralized training with decentralized execution and value decomposition assumption, SIEMA encourages multi-agent exploration and addresses the issue of low sample utilization through Intrinsic-Episodic Dual Exploration. The Cooperative Exploration Intrinsic Reward (CEIR) component incentivizes exploration from various aspects, incorporating novelty, optimal distance, and cooperative exploration. Episodic Exploration Replay (EER) explores at the episode level, ensuring optimal utilization of all samples in the replay buffer. Furthermore, we introduce the distributed scalable multi-agent training framework to accelerate the learning process and address the issue of low sample generation in MARL by deploying multiple workers and actors in a distributed manner. We illustrate the advantages of SIEMA by ablation experiments, and demonstrate its remarkable superiority over state-of-the-art MARL algorithms on challenging tasks in the StarCraft II micromanagement benchmark.