Reunion: Receiver-driven network load balancing mechanism in AI training clusters

Abstract

RDMA over Converged Ethernet (RoCEv2) enables high-performance networking for large-scale model training but faces challenges due to traffic characteristics such as elephant flows, low entropy, and traffic bursts. Conventional load-balancing techniques like ECMP struggle with hash collisions, causing increased tail latency. Advanced solutions, such as source routing and enhanced ECMP, mitigate these issues but still have hash collisions when there are multiple sources. While packet spraying and flow slicing help alleviate load imbalances due to hash collisions, they can intensify packet reordering issues. Reunion, a novel mechanism for RoCEv2 environments, tackles three critical challenges of flowlet-based rerouting: (1) Rerouting decisions introduce new hash conflicts; (2) The out-of-order packets caused by rerouting has a significant impact on small flows; (3) Setting appropriate flowlet timeout values in high-bandwidth environments is difficult. By utilizing Count-Min-Sketch to filter out small flows and aggregating real-time congestion data, Reunion enables source switches to make dynamic rerouting decisions for elephant flows, minimizing congestion hotspots. Simulations conducted using NS-3 highlight Reunion’s robustness and effectiveness in reducing tail latency. Under varying network loads, Reunion outperforms existing load-balancing schemes such as Conga, LetFlow, ECMP, and ConWeave, achieving tail latency reductions ranging from 10.9% to 62.1%.