Method for Enabling RDMA Transport Peer to Peer Transfer with NVMeoF Ethernet SSDs

Abstract

P2P (peer to peer) transfer enables to offload Host CPU & its DRAM, while running critical enterprise applications like replication & snapshot. P2P is inherently possible in the PCIe domain because of the Address based and ID based routing characteristics of the single root PCIe topology. CMB (controller memory buffer) can be utilized in PCIe based NVMe (Non-Volatile Memory Express) SSDs to achieve P2P transfer without involving the Host CPU & DRAM [1]. Now a days Data Centers trending to migrate to a scalable, disaggregated storage transports like NVMeoF (Non-Volatile Memory Express Over Fabrics) [4]. Moreover, peers may reside on different racks of Data Center connected with NVMeoF transport. Currently there is no inherent method available to use CMB feature [5] and doing P2P transfer in NVMeoF. In this paper we approach this problem by using VUC (vendor unique command) for sharing Peer's connection information & establishing a dedicated P2P QP (Queue pair) connection among the peer Ethernet SSDs. Through this method the host CPU will just give control messages to initiate the P2P transfer so that highly intensive IO transfer responsibility is fully offloaded to peer NVMeoF SSDs. The proposed approach is energy efficient and nullifies the power consumed by Host DRAM for data copy & reduces load on ethernet switch to half, while the data copy speed is doubled without affecting the throughput.