Adaptive transport service selection for MPI with InfiniBand network

Abstract

We propose a method which adaptively selects InfiniBand transport services used for source and destination peers to improve performance while limiting memory consumption of the MPI library. There are two major choices of IB transport services available, i.e., Reliable Connection (RC) and Dynamically Connected (DC), each of which is selected for each pair of source peer and destination peer. RC is faster than DC for all communication patterns except for the case where there are many active RCs. It also consumes a lot of memory when there are many processes. DC, on the other hand, consumes less memory than RC but its performance drops when sending messages to different destinations or when many DCs sends a message to the same destination DC. Therefore, the library should find the best mapping of RCs and DCs to pairs of source peer and destination peer according to the communication pattern of the application. Our method finds a good mapping by comparing potential latency benefits for candidate mappings. It achieves 13%--19% latency reduction when compared to the methods using only DCs in micro-benchmarks representing communication patterns problematic to RC or DC with 64 processes.