GPU Cluster RDMA communication technology and congestion control

Abstract. This paper discusses Remote Direct Memory Access(RDMA) communication technology and the congestion control methods for Graphics Processing Unit(GPU) clusters. The implementation methods of RDMA networks widely used in GPU clusters are studied. Three implementation modes including InfiniBand, iWARP, and RoCE are analysed with comparison of their performance and applicable environments. Then, based on the analysis of a new congestion controls algorithm, DBCC & CBFC algorithm, is proposed. This algorithm based on delay feedback control and credit flow control prevents network congestion or increased latency in GPU cluster RDMA networks. The working principles of the algorithm are introduced including calculating the adjustment amount of the sending rate, initializing the sender and receiver and mechanisms to handle packet loss and timeout. Experimental results show that the algorithm optimizes network performance with RDMA communication in GPU clusters, while avoiding congestion and minimizing packet loss. However, due to the limitation of experimental conditions, it is not possible to conduct more environmental tests. In practical application, the applicability of the algorithm needs to be carefully evaluated and adjusted according to the specific situations.