Skipping congestion-links for coflow scheduling

Abstract

Data transfer duration accounts for a great proportion of job completion time in big-data systems. To reduce the time spent on data transfer, some traffic scheduling mechanisms at coflow-level are proposed recently. Most of them abstract datacenter networks as an ideal non-blocking big-switch, and the bottleneck is located at egress or ingress ports of end-hosts instead of in networks. Thus, they mainly focus on how to allocate port capacities of end-hosts to jobs without considering in-network congestion. However, link congestion frequently occurs in datacenter networks due to network oversubscription and load imbalance. When link congestion occurs, bottleneck locations will move from the ports of end-hosts to network links. In this paper, we design and implement SkipL, a congestion-aware coflow scheduler which could detect congestion and schedules coflows at end-hosts to effectively reduce coflow completion time. In addition, to be easily deployed in cloud environments, SkipL does not require to control flow routes. SkipL prototype system is implemented in Linux. The results of experiments conducted in a real small testbed and simulations conducted in the flow-level simulator show that SkipL reduces the average Coflow Completion Time(CCT) compared to the per-flow fair sharing scheduling method and Varys.