High Performance RDMA-based Multi-port All-gather on Multi-rail QsNet II

Abstract

Scientific applications written in MPI use collective communications intensively. Efficient and scalable implementation of such collective operations is therefore crucial to the performance of MPI applications running on clusters. Quadrics QsNetII is a high-performance network that implements some collectives at its Elan user-level library. Its MPI implementation uses such primitives directly. Quadrics communication software supports point-to-point message striping over multi-rail QsNetII networks. However, multi-rail collectives, other than broadcast, are not supported. In this work, we propose, design and implement a number of RDMA- based multi-port algorithms for the all-gather operation over multi-rail QsNetII clusters directly at the Elan level. Our performance results indicate that the proposed multi-port all-gather Direct algorithm gains an improvement of up to 1.49 for 32 KB messages over the native Elan_gather() collective.