Mutual information oriented deep skill chaining for multi-agent reinforcement learning

Abstract

Multi-agent reinforcement learning relies on reward signals to guide the policy networks of individual agents. However, in high-dimensional continuous spaces, the non-stationary environment can provide outdated experiences that hinder convergence, resulting in ineffective training performance for multi-agent systems. To tackle this issue, a novel reinforcement learning scheme, Mutual Information Oriented Deep Skill Chaining (MioDSC), is proposed that generates an optimised cooperative policy by incorporating intrinsic rewards based on mutual information to improve exploration efficiency. These rewards encourage agents to diversify their learning process by engaging in actions that increase the mutual information between their actions and the environment state. In addition, MioDSC can generate cooperative policies using the options framework, allowing agents to learn and reuse complex action sequences and accelerating the convergence speed of multi-agent learning. MioDSC was evaluated in the multi-agent particle environment and the StarCraft multi-agent challenge at varying difficulty levels. The experimental results demonstrate that MioDSC outperforms state-of-the-art methods and is robust across various multi-agent system tasks with high stability.