Reducing Imbalance Ratio in MapReduce

Abstract

In order to speed up the processing, MapReduce invokes many mappers and reducers concurrently. Each mapper sends the intermediate map-outputs to reducers according to the key of data. For some big data with the property of data skew, some partitions will own a huge amounts of data. Thus, some reducers need more time to process their assigned partitions, resulting in increasing the total execution time. This paper proposes a balanced partition method to divide the intermediate map-outputs evenly. The balanced partition method has a preprocessing mapreduce (mapper1 and reducer1) by which partitioner is derived. The mapper1 is used to counting key frequencies by employing trie data structure efficiently. In reducer1, based on all the key frequencies, many sub-partitions are derived by cut-points and these sub-partitions are evenly distributed to partitions. The cut-points and the mapping table are used in every mappers of the application mapreduce for partitioning the intermediate map-outputs evenly, resulting in reducing the execution time.