WaxElephant: A Realistic Hadoop Simulator for Parameters Tuning and Scalability Analysis

Abstract

MapReduce is becoming the state-of-the-art computation paradigm for processing large-scale datasets on a large cluster with tens or thousands of nodes. Hadoop, an open-source implementation of MapReduce framework, has gained much popularity due to its high scalability and performance. Two challenging issues for a large-scale Hadoop cluster are how to analyze the scalability and identify the optimal parameters configurations. To address these issues, we designed and implemented a Hadoop simulator called Wax Elephant, which provides the following capabilities: (1) loading real MapReduce workloads derived from the historical log of Hadoop clusters, and replaying the job execution history, (2) synthesizing workloads and executing them based on statistical characteristics of workloads, (3) identifying the optimal parameters configurations, and (4) analyzing the scalability of the cluster. Extensive experiments have been conducted to validate the accuracy of the Wax Elephant simulator.