Scheduling Coflows by Online Identification in Data Center Network

Abstract

Recently, many scheduling schemes leverage coflows to improve the communication performance of jobs in distributed application frameworks deployed in data center networks, such as MapReduce and Spark. Most of them require application modification to obtain the coflow information such as the coflow ID. The latest work CODA suggests non-intrusively extracting coflow information via an identification method. However, the method depends on the historical traffic information, which may cause the identification accuracy to decrease a lot when traffic varies. To tackle the problem, we present SOCI for Scheduling coflows by the Online Coflow Identification. By observing that flows in a coflow typically communicate with a master process for starting and ending in the up-to-date distributed application frameworks, SOCI uses this characteristic for the online coflow identification. Given identification errors are inevitable, the coflow scheduler in SOCI adopts a Selectively Late Binding (SLB) mechanism, which associates the misclassified flows with coflows according to the estimation on the impact of this association on the average Coflow Completion Time (CCT). The trace-driven simulations show that SOCI can reduce CCT by up to $1.23\times$ compared to CODA when the identification accuracy decreases and is comparable to schemes without coflow identification.