SWIFT: A Transparent and Flexible Communication Layer for PCIe-Coupled Accelerators and (Co-)Processors

Abstract

The Peripheral Component Interconnect Express (PCIe) is the predominant interconnect enabling the CPU to communicate with attached input/output and storage devices. Considering its high performance and capabilities to connect different address domains via the so-called Non-Transparent Bridging (NTB) technology, it starts to be an alternative or addition to traditional interconnects. The PCIe technology enables devices to communicate in a peer-to-peer manner allowing for new implementation possibilities of tomorrow's high-performance systems. Components being attached to the same computer rack are connected by means of PCIe and the racks themselves by using traditional network technologies. This leads to a heterogeneous landscape of compute nodes and high-performance interconnects. The Socket Wheeled Intelligent Fabric Transport (SWIFT) takes up the challenge of programming these systems. The presented implementation is highly portable due to a hardware abstraction layer allowing for bringing the implemented concepts to new interconnects with minimal effort. It is evaluated on a test system exposing different compute nodes equipped with coprocessors, which take part in a PCIe non-transparent bridging architecture. Besides low-level benchmarks investigating principal performance characteristics of the communication layer, MPI benchmark results are presented illustrating how scientific applications may be ported to heterogeneous environments.