Optimal All-to-All Personalized Communication in All-Port Tori

Abstract

All-to-all personalized communication, also known as complete exchange, is one of the most dense communication patterns in parallel computing. In this paper, we propose new indirect algorithms for complete exchange on all-port ring and tori. The new algorithms fully utilize all communication links and transmit messages along shortest paths to completely achieve the theoretical lower bounds on message transmission, which has not been developed among other existing indirect algorithms. For 2D rtimesc (rlesc) all-port torus, the algorithm has time complexities of optimal transmission cost and O(c) startup cost. The algorithm for 4D torus follows a similar structure. In addition, the proposed algorithms accommodate non-power-of-two tori where the number of nodes in each dimension needs not be power-of-two or square. Finally, the algorithms are conceptually simple and symmetrical for every node so that they can be easily implemented and achieve the optimum in practice