vClos: Network contention aware scheduling for distributed machine learning tasks in multi-tenant GPU clusters

Abstract

Distributed machine learning (DML) technology enables training large neural networks in reasonable time. However, with computing power growing faster than network capacity, network communication becomes the DML training bottleneck. Current multi-tenant GPU clusters suffer network contention due to hash-collision, which not only increases the overhead of communication but also will increase the waiting time of users. This paper analyzes how network contention slows training throughput and summarizes training traffic patterns as a $P$-incast-free Pattern. We propose a Balanced Routing which leverages training traffic patterns to reduce contention. Furthermore, we introduce vClos to handle contention through jointly considering topology, routing, communication pattern, and GPU assignment. Evaluations via testbed experiments and real-trace-based simulations show vClos reduces average job completion time up to 67.6% compared to ECMP in heavy workloads.