SwiftS: A Dependency-Aware and Resource Efficient Scheduling for High Throughput in Clouds

Abstract

An increasing number of large-scale data analytics frameworks moves towards larger degrees of parallelism aiming at high throughput guarantees. It is challenging to design a scheduler with high throughput and high resource utilization due to task dependency and job heterogeneity. The state-of-the-art schedulers in cloud/datacenters cannot well handle the scheduling of heterogeneous jobs with dependency constraints (e.g., dependency among tasks of a job) for simultaneously achieving high throughput and high resource utilization. We propose SwiftS: a dependency-aware and resource efficient scheduling for high throughput in clouds.