Multi-Agent Reinforcement Learning Investigation Based on Football Games

Games are classic scenarios for reinforcement learning, and the support of a variety of standard tasks and experimental platforms is one of the reasons for the success of reinforcement learning. In the actual environment, interacting with other individuals is often necessary to be considered, such as in games Go, Poker, football and tennis, which require multi-agent participation. This paper implemented some multi-agent investigation based on Google Research Football to show the performance gap between the centralized trained policy, which produces one action when called; and a unified “super agent”, which produces multiple actions simultaneously. Then it compared the sensitivity of the learned policies by perturbing the policy parameters / observations / initial states / cooperative agent. In this case, Proximal Policy Optimization (PPO) method has faster convergence than IMPALA, and using mean provides higher episode mean reward than using the max operator for aggregation. Besides, a multi-player experiment was also tried out, the results show that the less the episode difference is, the greater the expected return it gets.