LBVP: A load balance algorithm based on Virtual Partition in Hadoop cluster

Abstract

An approach based on Virtual Partition is proposed to improve the load balance in Reduce phase in MapReduce-based system in cloud computing. After each Map task finished, the output keys are partitioned to different virtual partitions according to Hash Function. And LBVP (a load balance algorithm based on continuous virtual partition) is designed to combine all virtual partitions to the same number of reduce tasks, and ensure each reduce task having balanced input data. The experimental results indicate that the load balance of the amount of Reduce function input is improved effectively and the performance is not degraded significantly by using virtual partition and load allocation algorithm.