Evolutionary Approach based Scheduler for Speculative Task Execution

Hadoop is an open source framework to implement MapReduce. It stores and processes the data in distributed, highly scalable, parallel and fault tolerant environment. Job scheduling shows a significant role in optimizing the functioning of Hadoop. Hadoop default scheduler is not suitable for heterogeneous environment and not robust to identify the stragglers task which prolongs total execution time. Evolutionary approach based scheduler for speculative task execution is proposed in this paper. In this work we are proposing a new method to select the best nodes to run the speculative copy of the slow task. Two parameters such as network information and resource utilization are used to select the optimal nodes to execute the speculative copy of the stragglers task. Experiments have been conducted on web log file of academic website for obtaining the click count. Experimental results show that the execution time is reduced by 31% for 1 GB input data and 23% for 2 GB input data. On an average, the execution time is improved by 21% compared to conventional scheduler.