P-Socket: optimizing a communication library for a PCIe-based intra-rack interconnect

Data centers require efficient, low-cost, flexible interconnects to manage the rapidly growing internal traffic generated by an increasingly diverse set of applications. To meet these requirements, data center networks are increasingly employing alternatives such as RapidIO, Freedom, and PCIe, which require fewer physical devices and/or have simpler protocols than more traditional interconnects. These networks offer raw high performance communication capabilities, but simply using them for conventional TCP/IP-based communication fails to realize the potential performance of the physical network. Here we analyze causes for this performance loss for the TCP/IP protocol over one such fabric, PCIe, and we explore a hardware/software solution that mitigates overheads and exploits PCIe's advanced features. The result is P-Socket, an efficient library that enables legacy socket applications to run without modification. Our experiments show that P-Socket achieves an end-to-end latency of 1.2μs and effective bandwidth of up to 2.87GB/s (out of a theoretical peak of 3.05GB/s).