Optimizing global parameter synchronization for geo-distributed machine learning in reconfigurable optical wide area networks

Abstract

Geo-distributed machine learning (Geo-DML) usually uses a hierarchical training architecture, local parameter synchronization (LPS) within data center and global parameter synchronization (GPS) between data centers. Compared to fast LAN bandwidth, the heterogeneous and scarce WAN bandwidth becomes one of the main bottlenecks of training performance for Geo-DML. Fortunately, the emerging optical technologies render the modern WAN topology reconfigurable, which has been adopted to improve the performance of some traditional traffic with the help of software-defined networking (SDN). However, the reconfigurable WAN topology is often overlooked by most schemes aimed at accelerating Geo-DML. In this paper, we propose AdaptivePS, an adaptive global parameter synchronization scheduling scheme that leverages the reconfigurable feature of WAN topology and the training characteristics to speed up Geo-DML training. Specifically, mathematical optimization models considering the topology construction and parameter synchronization scheduling are firstly established. Then AdaptivePS solves the mathematical models through relaxing and deterministic rounding scheme, obtaining the deployment of global aggregation nodes, wavelength allocation, path and rate allocation. The simulation results based on real WAN topologies show that compared to RoWAN, RAPIER and Baseline, AdaptivePS can reduce global communication time (GCT) by up to $73.4\%$, $86.7\%$, $96.2\%$, respectively. This demonstrates that AdaptivePS can effectively cope with different network environments, with the help of adaptive selection of global aggregation nodes, reconfigurable topology, and mathematical model based scheduling.