PIFE: Permutation Invariant Feature Extractor for Danmaku Games

Abstract

Dealing with unstructured complex patterns provides a challenge to existing reinforcement patterns. In this research, we propose a new model to overcome the difficulty in challenging danmaku games. Touhou Project is one of the bestknown games in the bullet hell genre also known as danmaku, where a player has to dodge complex patterns of bullets on the screen. Furthermore, the agent needs to react to the environment in real-time, which made existing methods having difficulties processing the high-volume data of objects; bullets, enemies, etc. We introduce an environment for the Touhou Project game‘東方花映塚~Phantasmagoria of Flower View.’ which manipulates the memory of the running game and enables to control the character. However, the game state information consists of unstructured and unordered data not amenable for training existing reinforcement learning models, as they are not invariant to order changes in the input. To overcome this issue, we propose a new pooling-based reinforcement learning approach that is able to handle permutation invariant inputs by extracting abstract values and merging them in an order-independent way. Experimental results corroborate the effectiveness of our approach which shows significantly increased scores compared to existing baseline approaches.