Improved Adaptive Feedback Scheduling Algorithm based on LATE in Hadoop Platform

Abstract

Hadoop is the mainstream cloud platform for data analysis and processing. Job scheduling algorithm directly affects job response time and system resource utilization. The research and improvement of scheduling algorithm has always been an important topic. Based on the original LATE scheduling algorithm, this paper proposes an adaptive feedback LATE (AF-LATE) algorithm to improve the autonomous selection and feedback of execution nodes and backup tasks. In the process of scheduling, according to the load type of the task, the idle node with highest ratio of task success rate to node load is selected to back up the backward task. At the same time, the feedback of the task and node working data is obtained to dynamically adjust the fast and slow node set. The algorithm improves the resource utilization and load balance, and improves the reliability of task execution and reduces the running time of scheduling algorithm. In this paper the experimental environment is built to verify the algorithm. The results show that the scheduling algorithm is more reasonable in judging backward tasks and selecting execution nodes in heterogeneous environment, which can shorten the response time of jobs, improve the utilization and efficiency of the cluster, and can adaptively adjust the performance of execution nodes to improve the cluster reliability.