A Simulation for Forecasting Compute Resource Usage

Abstract

The usage of compute resources by data processing jobs may change over time, requiring careful resource planning when an organization operates these resources itself in an on-premise private cloud. Ideally, the currently available resources always match the need of jobs executed on them. This way the resources would neither be overutilized, which is usually undesirable as the jobs might take longer, nor underutilized, which causes unnecessary costs for unused resources. When an organization decides to extend its private cloud resources, it can still take months until the servers are bought, delivered, and installed. Thus, the resources have to be planned carefully in advance. Estimating the future resource needs is difficult and influenced by many factors. In our experience, creating the estimate is often a manual process supported by self-designed spreadsheets; these spreadsheets are maintained by a single person from time to time and might even be replaced completely if someone else assumes that person's responsibility. However, this approach does not lead to transparent and verifiable forecasts that enable collaboration and learning from past decisions. This paper addresses the problem of generating a transparent and verifiable compute resource usage forecast by proposing a simulation approach. It requires a user to model an estimate of the future workload development of the data processing jobs as well as the current compute resource setup. The simulation can then be run to identify possible future resource bottlenecks. This can be repeated for different scenarios, including situations of failing resources as well as the addition of resources to compensate for bottlenecks and failures. We further provide a first qualitative case study of this approach that demonstrates its potential.