Using Reinforcement Learning to Generate Levels of Super Mario Bros. With Quality and Diversity

Abstract

Procedural content generation (PCG) is essential in game development, automating content creation to meet various criteria such as playability, diversity, and quality. This article leverages reinforcement learning (RL) for PCG to generate Super Mario Bros. levels. We formulate the problem into a Markov decision process (MDP), with rewards defined using player enjoyment-based evaluation functions. Challenges in level representation and difficulty assessment are addressed by conditional generative adversarial networks and human-like artificial intelligence agents that mimic aspects of human input inaccuracies. This ensures that the generated levels are appropriately challenging from human perspectives. Furthermore, we enhance content quality through virtual simulation, which assigns rewards to intermediate actions to address a credit assignment problem. We also ensure diversity through a diversity-aware greedy policy, which chooses not-bad-but-distant actions based on $Q$ -values. These processes ensure the production of diverse and high-quality Super Mario levels. Human subject evaluations revealed that levels generated from our approach exhibit natural connection, appropriate difficulty, nonmonotony, and diversity, highlighting the effectiveness of our proposed methods. The novelty of our work lies in the innovative solutions we propose to address challenges encountered in employing the PCG via RL method in Super Mario Bros. , contributing to the field of PCG for game development.