Performance Modeling and Task Scheduling in Distributed Graph Processing

Abstract

The accelerated growth of datasets observed in modern applications also applies to datasets modeled as graphs. To handle this problem, several large scale distributed graph processing models have been proposed, such as Pregel. These systems are designed to run in large clusters, where the resources must be allocated efficiently. In this paper we present a prediction model and a scheduler for Pregel-based distributed graph processing jobs. The jobs are treated as moldable tasks by the scheduler that, based on the predictions, allocates the best number of workers to each job in order to minimize makespan. Experimental results show that the prediction model has accuracy close to 90%, allowing the scheduler to work within the theoretical approximation limits of the optimal makespan.