Smart-mDAG: An Intelligent Scheduling Method for Multi-DAG Jobs

Abstract

Job scheduling is a fundamental problem in cloud data center, which plays an essential role in the makespan, resource utilization and maintenance of scheduling security, it has received widespread attention. With the rapid increase of jobs' amount, higher requirements are put forward for scheduling efficiency and makespan. Meantime dependencies between tasks are closely related to makespan and throughput, and these dependent tasks form multiple DAG structures. Heuristic algorithms have limitation on adjusting the scheduling policy according to diverse dependencies, thus resulting the extension of makespan. In this paper, we propose an intelligent scheduling method for multi-DAG jobs using deep reinforcement learning, called Smart-mDAG. It is a job-specific scheduling method that adjusts the scheduling policy based on diverse dependencies to minimize the makespan. Firstly, we convert dependencies to numeric form through a feature extraction module to obtain the dependent information from the DAG. Secondly, we use cascaded neural networks to implement the fusion of scheduling information, so we can obtain the fitness between machines and tasks. With Alibaba Cluster Data V2018, we evaluate the performance of Smart-mDAG on a five-machines small cluster. The result shows that compared to control algorithms, Smart-mDAG can shorten the makespan for 70% jobs, and the optimal makespan of single job can be decreased to 65% of the past.