Decoupling Control and Data Transmission in RDMA Enabled Cloud Data Centers

Advances in storage, processing, and networking hardware are changing the structure of distributed applications. RDMA networks provide multiple communication mechanisms that enable novel hybrid protocols specialized to different data transfer requirements. In this paper, we present a distributed communication scheme that separates control and data communication channels directly at the RNIC rather than the application level. We develop a new communication artifact, a remote random access buffer, to efficiently implement this separation. Data messages are sent silently to the receiver, which is informed of the location of the data by a subsequent control message. Experiments on an RDMA-enabled cluster with micro benchmarks and two distributed applications validate the performance benefits of our approach.