PAARD: Proximity-aware all-reduce communication for dragonfly networks

Abstract

The All-Reduce operation is one of the most widely used collective communication operations, and it is widely used in the research and engineering fields of high-performance computing(HPC) and distributed machine learning(DML). Previous optimization work for All-Reduce operation is to design new algorithms only for different message size and different number of processors, and ignores the optimization that can be achieved by considering the topology. Dragonfly is a popular topology for current and future high-speed interconnection networks. The hierarchical characteristics of dragonfly network can be utilized to effectively reduce hardware overhead while ensuring low end-to-end transmission latency. This paper offers a first attempt to design an efficient All-Reduce algorithm on dragonfly networks, referenced as PAARD. Based on the hierarchical characteristics of dragonfly network, PAARD first proposes an end-to-end solution to alleviate congestion that could remarkably boost performance. We carefully design the algorithm of PAARD to ensure desirable performance with acceptable overhead and guarantee the generality when met marginal cases. Then, to illustrate the effectiveness of PAARD, we analyze the performance of PAARD with the state-of-the-art algorithm, Halving-doubling(HD) algorithm and Ring algorithm. The simulation results demonstrate that in our design the execution time can be improved by 3X for HD and 4.19x for Ring on 256 nodes of a 342-node dragonfly with minimal routing.