Model Representation Considerations for Artificial Intelligence Opponent Development in Combat Games

Abstract

The performance and behavior of an Artificial Intelligence (AI) opponent in games requires coordination of multiple agents and complex tasks depends on many design choices made during implementation. Currently, gaming agents developed with Reinforcement Learning (RL) methods are constructed to play the game, leading to natural design choices for observations, actions, and rewards that are congruent with a human player's actions and objectives. However, in simulation and serious games, the objective of the implemented opponent should be developed in a way that supports the learning objectives for the user, such as by including additional ground truth environment data in the observation space or action structure. Therefore, the reward structure for the AI needs to incorporate more sophisticated considerations than just whether the game was won or lost by the AI. In this way the design space for opponent AI in these settings is considerably broader than what is traditionally used for RL gaming AI. This paper considers the implications of observation representation and reward design for the AI agent and associated actions in the context of 2-player battlefield-type games that are not strictly zero-sum. Semi-cooperative and fully competitive models are considered. The environment in these games is a spatially extended battlefield in which agents must maneuver their forces to bring them into combat range of each other. The objective of the game is control over a pre-specified location in the game, and combat is executed via Lanchester attrition. We demonstrate the impact of aggregation on stochasticity of the model, where aggregation of the state model is controlled by various entropy-based metrics, as well as on the policy learned by an RL agent. Generalizations to alternative scenarios and objectives are discussed, as well as applications to the development of an AI combat opponent that can cohesively manage its forces over multiple scales.