A Dynamic Resource Allocation Framework for Apache Spark Applications

Abstract

In this paper we design and implement a middleware service for dynamically allocating computing resources for Apache Spark applications on cloud platforms, and consider two different approaches to allocate resources. In the first approach, based on limited execution data of an application, we estimate the amount of resource adjustment (i.e., Delta) for each application separately a priori which is static during the execution of that particular application (i.e., Approach - I). In the second approach, we adjust the value of Delta dynamically during runtime based on execution pattern in real-time (i.e., Approach - II). Our evaluation using six different Apache Spark applications on both physical and virtual clusters demonstrates that our approaches can improve application performance while reducing resource requirements significantly in most cases compared to static resource allocation strategies.