Methodology to Evaluate the Performance of Hadoop MapReduce on a Hyper-V Cluster using SAN Storage

Abstract

Deploying Hadoop MapReduce applications in a virtualized environment is adopted by some cloud computing providers for better resource utilization. However, the virtualization overhead can negatively affect the performance of applications when executed on virtual machines rather than physical servers. In that regard, this paper introduces a methodology to match the software and hardware specifications of virtual and physical Hadoop clusters to allow accurate measurement of virtualization overhead as well as enable the efficient execution of MapReduce applications on both clusters. The methodology considers configuring non uniform memory access in the utilized powerful servers. It also factors in multipath aggregation that facilitates load balancing and failover protection when the servers access SAN storage over a local Ethernet network. The WordCount application with workloads that reached 750 GB was executed on both clusters to evaluate the effects of virtualization on application performance. The results showed that the average elapsed time of a MapReduce application on a specific virtual cluster can be 19.2% higher than that on a physical cluster, mainly due to I/O throughput degradation over the virtual cluster in that case.