Improving Energy Efficiency of IO-Intensive MapReduce Jobs

Abstract

Map-Reduce is a popular data-parallel programming model for varied analysis of huge volumes of data. While a multicore and many CPU HPC infrastructure can be used to improve parallelism of map-reduce tasks, IO-bandwidth limitations may make them ineffective. IO-intensive activities are essential in any MapReduce cluster. In HPC nodes, IO-intensive jobs get queued at the IO-resources while the CPU remain underutilized, resulting in a poor performance, high power consumption and thus, energy inefficiency. In this paper, we investigate which power management setting can be used to improve the energy efficiency of IO-intensive MapReduce jobs by performing a thorough empirical study. Our analysis indicates that a constant CPU frequency can reduce the energy consumption of an IO-intensive job, while improving its performance. Consequently, we build a set of regression models to predict the energy consumption of IO-intensive jobs at a CPU frequency for a given input data volume. We obtained same set of models, with different coefficients, for two different types of IO-intensive jobs, which substantiates the suitability of identified models. These models predict respective outcomes with 80% accuracy for 80% of the new test cases.