Enhancing imitation learning training for non-player characters based on provenance data

Abstract

Non-Player Characters (NPCs) play a crucial role in the immersive experience of a game world. When designed effectively, NPCs have unique personalities and react realistically to player actions. Meeting players’ expectations for NPCs to resemble real individuals has become a major focus for game developers striving to enhance immersion.

In this work, we propose the use of data collected via provenance to create a model for training an NPC to act similarly to a human player using Imitation Learning. The main goal of this work is to improve the training efficiency of the agent, while preserving the high level of believability achieved in previous work. To this end, provenance is employed not only as a form of logging, but also as a means to guide and optimize the learning process — a contribution not previously explored in the literature.

To validate our model, we used the DodgeBall game within the Unity ML-Agents Toolkit for the Unity Engine. We compared our trained agent with an agent from previous work that used provenance solely for logging. Using win rate as a proxy for training efficiency, agents trained with our new model outperformed those trained with the previous approach, when evaluated after the same number of training steps.

Additionally, we created scenarios in which players participated in matches against both the new and previous agents, rating their believability. The results were promising in terms of both perceived believability and the efficiency of the training process.

In this work we propose to use data collected via provenance to create a model for training an NPC to act similarly to a human player using Imitation Learning. We use provenance not only as a form of log, but also to improve training efficiency, something that has not been presented in the literature until now. To validate our model, we used the DodgeBall game within the Unity ML-Agents Toolkit for Unity Engine. We compared our trained agent with an agent trained in previous work, which use provenance as a form of logging. Through matches between the two agents, those that were trained with our new model demonstrated greater efficiency. Additionally, we created scenarios of players participating in games against our current agent and our previous solutions, rating the believability of each. The results were quite promising, both in terms of believability and training efficiency.