Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

Abstract

This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.