Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

Abstract

One of the most popular types of online games is the Minecraft-like Game (MLG), in which players can terraform the environment. MLGs currently support their many players by replicating isolated instances with limited scalability. We posit that performance variability is a key cause for the lack of scalability in MLGs and design the first benchmark that focuses on MLG performance variability, identifying specialized workloads, metrics, and processes. We conduct real-world benchmarking of MLGs, both cloud-based and self-hosted. We find environment-based workloads and cloud deployment are significant sources of performance variability: peak-latency degrades sharply to 20.7 times the arithmetic mean, and exceeds by a factor of 7.4 the performance requirements.