An Efficient Dynamic Load-Balancing Large Scale Graph-Processing System

Abstract

Since the introduction of pregel by Google, several large-scale graph-processing systems have been introduced. These systems are based on the bulk synchronous parallel model or other similar models and use various strategies to optimize system performance. For example, Mizan monitors the workload of each worker to determine whether the workload between the workers is balanced with respect to the execution time. If the workload is unbalanced among workers, Mizan migrates nodes from overloaded workers to under-loaded workers to balance the load among workers and minimize the total execution time. On the basis of Mizan's migration plan, we implement a graph-processing system called GPSer with an efficient re-partitioning graph scheme. Our system uses statistical tools, e.g., coefficient of variation and correlation coefficient, to modify the migration plan and determine whether the workloads are balanced among all workers. Our system can accurately monitor current work-loads and decide whether to migrate nodes among workers to balance the load. When imbalance arises, the work-load of all workers can quickly converge to a balanced state, thereby enhancing the system performance. In experiment our system outperforms the state-of-the-art dynamic load-balancing graph processing-system, such as Mizan.