Low Latency, High Bisection-Bandwidth Networks for Exascale Memory Systems

Abstract

Data movement is the limiting factor in modern supercomputing systems, as system performance drops by several orders of magnitude whenever applications need to move data. Therefore, focusing on low latency (e.g., low diameter) networks that also have high bisection bandwidth is critical. We present a cost/performance analysis of a wide range of high-radix interconnect topologies, in terms of bisection widths, average hop counts, and the port costs required to achieve those metrics. We study variants of traditional topologies as well as one novel topology. We identify several designs that have reasonable port costs and can scale to hundreds of thousands, perhaps millions, of nodes with maximum latencies as low as two network hops and high bisection bandwidths.