Argus: Efficient Job Scheduling in RDMA-assisted Big Data Processing

Abstract

Efficient job scheduling is an important and challenging issue in big data processing systems. Traditional designs commonly give priority to data locality during scheduling and follow a network-optimized principle to avoid costly data moving across the network. The emergence of the high-performance Remote Direct Memory Access (RDMA) network brings new opportunities for big data processing systems. However, the existing RDMA-assisted designs ignore the dependency among stages during scheduling and this can result in unsatisfied system efficiency. In this work, we propose Argus, a novel RDMA-assisted job scheduler which achieves high resource utilization by fully exploiting the structure feature of stage dependency. Argus prioritizes the stages whose completion can enable more schedulable stages. We implement Argus on top of RDMA-Spark, and conduct comprehensive experiments to evaluate the performance using large-scale traces collected from real-world systems. Results show that compared to state-of-the-art designs, Argus reduces the job completion time and makespan by 38% and 31%, respectively.