IP-NUMA for low-latency communication

Abstract

With cloud service becoming more popular, low-latency communication is required between servers in a data center. Low-latency node-to-node or application-to-application notification can be achieved in a NUMA [1] (Non-Uniform Memory Access) system, but requires dedicated, special-purpose network infrastructure. However, IP networks are commonly used in data centers. Using a custom FPGA-based NIC, we create NUMA-like behavior using hardware-generated IP packets which can be transmitted over commodity Ethernet switches and IP routers. We demonstrate ping-pong acknowledgments between two PCs equipped with our IP-NUMA boards. Our IP-NUMA implementation exhibits latency as much as ten times lower than software using Berkeley sockets over a consumer-grade Ethernet switch. iSCSI initiator-target communications and transaction-based distributed software systems will benefit from the reduced latency.