Performance Evaluation of Load Balancing Algorithms in Hadoop

Abstract

Hadoop is a popular model used in shared-nothing clusters for data-intensive parallel computing. The MapReduce-algorithm is a model that operates on distributed, parallel systems. Hadoop has different implementation of this MapReduce-algorithm. Some of the implementations during execution produce an imbalance of work on the cluster. The performance of MapReduce mainly depends on data distribution which is one of the main issues as the load is not balanced among nodes. FIFO job scheduler that serves the jobs in their submission order is used by MapReduce to balance the load but unfortunately it is inefficient in real world cases as it missed many important factors that impact the performance such as heterogeneity factor and data skewness, so Load balancing is important to make all resources utilized evenly and more efficiently. Load balancing is an approach of improving the system’s performance by redistributing the load among nodes.The main goal of this work is to execute various load balancing algorithms in hadoop framework. In this dissertation various load balancing algorithms such as Randomized Hydrodynamic Load Balancing, Cogset Load Balance, Block-based Load Balancing for Entity Resolution, Ant colony Optimization, Shortest Path are simulated and comparisons being made on the basis of various parameters like delay time, response time, throughput, turnaround time and threshold to find the best that solve the data distribution problem.