Cooperative Multi-agent Inverse Reinforcement Learning Based on Selfish Expert and its Behavior Archives

Abstract

This paper explores the multi-agent inverse reinforcement learning (MAIRL) method which enables the agents to acquire their cooperative behaviors based on selfish expert behaviors (i.e., it is generated from the viewpoint of a single agent). Since such selfish expert behaviors may not derive cooperative behaviors among agents, this paper tackles this problem by archiving the cooperative behaviors found in the learning process and by replacing the original expert behaviors with the archived one at a certain interval. For this issue, this paper proposes AMAIRL (Archive Multi-Agent Inverse Reinforcement Learning). Through the intensive simulations of the maze problem for our method, the following implications have been revealed: (1) AMAIRL is superior to MaxEntIRL in terms of finding cooperative behavior; (2) AMAIRL requires a long interval period to acquire the cooperative behaviors. In particular, AMAIRL with the long interval can find the cooperative behaviors that are hard to be found in AMAIRL with the short interval.