CD-CBA: a cross-domain communication-bound-aware resource allocation framework for pipeline-parallel distributed LLM training over dynamic multi-datacenter optical networks

Abstract

The explosive development in artificial intelligence (AI) like large language models (LLMs) requires massive graphics processing units (GPUs) during training, making distributed machine learning (DML) across multiple data centers (DCs) a practical approach to obtaining sufficient training resources. Among cross-DC DML techniques, pipeline parallelism (PP) is a natural solution, which partitions LLM layers across GPUs and orchestrates point-to-point communication between adjacent nodes. However, the high communication latency induces severe GPU idle time (“bubble”) due to poor communication and computation time overlapping in inter-DC PP. While prior works have explored partitioning and reordering computation tasks, the resource allocation for inter-DC communication between pipelined computing tasks over optical networks has been overlooked, resulting in higher GPU idle time and request congestion. Therefore, it is extremely important and challenging to propose an advanced inter-DC network resource allocation solution under the complex PP workload. In this work, we propose a cross-domain communication-bound-aware (CD-CBA) PP scheduling framework for multi-DC optical networks. Simulation results show that the proposed framework lowers the LLM training iteration time by up to 31.25%, decreases the bubble ratio by up to 11.81%, and reduces the transmission request blocking by up to 19.33%.