Software-defined cloud–optical networks for long-haul geographically distributed machine learning

Abstract

Optical networks enable long-haul geographically distributed machine learning (GDML) by connecting multiple data centers (DCs), offering a solution to overcome limitations of single DC-based training for large models. However, effective coordination is hindered by limited resource sharing among cloud and network entities. In this work, we propose an architecture of a software-defined cloud–optical network (SD-CON). Domain controllers of SD-CON jointly abstract cloud and network resources, while a hyper-domain controller establishes cloud–network service function chains (CN-SFCs) to enhance the cloud–network collaboration. Additionally, we introduce the task scheduling algorithm with a multi-candidate parameter server (MPS) to optimize the CN-SFCs. A 1000 km GDML experiment on the China Environment for Network Innovation demonstrates rapid allocation of cloud and network resources (${\sim}{5.7}\;{\rm s}$ latency) in SD-CON, improving task success rates (over 23.111%) and enhancing resource utilization compared with the baselines.