Developing game AI agent behaving like human by mixing reinforcement learning and supervised learning

Artificial intelligence (AI) agent created with Deep Q-Networks (DQN) can defeat human agents in video games. Despite its high performance, DQN often exhibits odd behaviors, which could be immersion-breaking against the purpose of creating game AI. Moreover, DQN is capable of reacting to the game environment much faster than humans, making itself invincible (thus not fun to play with) in certain types of games. On the other hand, supervised learning framework trains an AI agent using historical play data of human agents as training data. Supervised learning agent exhibits a more human-like behavior than reinforcement learning agents because of imitating training data. However, its performance is often no better than human agents. The ultimate purpose of AI agents is to entertain human players. A good performance and a humanlike behavior are important factors of the AI agents, and both of them should be achieved simultaneously. This study proposes frameworks combining reinforcement learning and supervised learning and we call then separated network model and shared network model. We evaluated their performances by the game scores and behaviors by Turing test. The experimental results demonstrate that the proposed frameworks develop an AI agent of better performance than human agent and natural behavior than reinforcement learning agents.