RECCL: optimizing collective algorithms for reconfigurable optical networks

With the introduction of optical circuit switching, network topologies in distributed machine learning systems can be periodically reconfigured. Existing collective strategies use fixed communication steps and relationships during model training, limiting their ability to fully utilize network resources. Therefore, we propose RECCL, a reconfigurable collective communication library that dynamically adjusts communication relationships in collective algorithms based on network topology reconfigurations, thereby enhancing link utilization. RECCL considers the interactions between collectives and reconfigurable networks and develops a collective cost model for various topologies by using the innovative collective sketch and fine-grained network model. Based on this cost model, RECCL designed an optimization algorithm that rapidly reconfigures collectives with minimal communication cost for the current topology, enabling reconfiguration for kilo-scale nodes within 300 ms. Our experiments show that RECCL’s reconfigured collectives are $1.08{-}2.5 \times$ faster than those in MPI or NCCL. RECCL can accelerate end-to-end training of GPT-2 and BERT by $1.12{-}1.65 \times$ for different batch sizes.