Experimental Analysis of InfiniBand Transport Services on WAN

Abstract

InfiniBand architecture (IBA) has emerged as a standard system-area interconnect in industry for both data-center and high performance computing. While IBA continues to evolve with more capabilities, it has yet to permeate the field of grid and pervasive computing across wide area. This requires a software infrastructure to take the full advantage of IBA networking mechanisms without being dwarfed by the long distances. In this paper, we present a detailed analysis of InfiniBand transport services for their suitability to data transfer and message passing in the wide area. Three of the common InfiniBand transport services have been examined, including reliable connection (RC), unreliable connection (UC), and unreliable datagram (UD). Our analysis indicates that UC and UD are better suited to provide high bandwidth for MPI on wide-area networks (WAN). We have also demonstrated that the current existing MPI implementations, designed over RC and UD, can be tuned to provide improved MPI bandwidth on WAN. Furthermore, by developing a UC-based MPI implementa-tion, we have shown that, at a distance of 8600 miles, MPI over UC can improve MPI bandwidth by as much as 100%.