Seagull -- A Real-Time Coflow Scheduling System

Abstract

Data-parallel applications often generate hundreds of flows at the same time in data centers. Since these flows are always connected with application context, traditional flow-level optimization policies are hard to perform well in such collections. The coflow abstraction brings hope and opportunity to make the scheduling much more efficient. But exsiting schedule systems based on that related concept are either static (such as Varys) or impracticable (such as Baraat). In this paper, we address these limitations by presenting Seagull -- a dynamic precise coflow scheduling system to optimize the average CCT (Coflow Completion Time) and guaranteeing predictable completions within coflow deadlines. It's a centralized system which can share the bandwidth resources with background flows in the data center. Our experiments show that 80% CCT of the coflows is about 1.7× faster than Varys. As for deadline meeting, Seagull can guarantee about 50% of admitted coflows finishing within their deadline, which is 10% more precise than Varys.