AITurbo

Abstract

As the scale and complexity of deep learning models continues to grow, model training is becoming an expensive job and only a small number of well-financed organizations can afford. Are the resources in commodity clusters well utilized for training? or how much potential space are still there for further improving the training efficiency in commodity clusters? is an urgent question to answer. In this paper, we review the processing of distributed learning training (DDL) in commodity GPU clusters and find that the current resource utilization is not only low but also imbalanced. We observe two features that can be exploited for further improving the training efficiency: partial predictable training and unified CPU-GPU training. Based on the observations, we present AITurbo, a novel resource scheduler that treats predictable and unpredictable jobs separately, but allocates heterogeneous CPU-GPU resource in a unified way. For predictable jobs, AITurbo designs a predicting model to estimate their performance under various heterogeneous resource allocations. For unpredictable jobs, it schedules them following the least-attained-service-first manner. AITurbo further designs a Borda-count based multi-level feedback queue method to combine them together. AITurbo demonstrates that there is still significant space for improving the training efficiency in commodity clusters. We evaluate AITurbo using jobs from Tensorflow benchmarks, which are submitted following the real trace of three production systems. Experimental results show that, compared with the state-of-the-art, AITurbo can reduce the average job completion time of DDL jobs by 3x.