Power-efficient networking for balanced system designs: early experiences with PCIe

Recent proposals using low-power processors and Flash-based storage can dramatically improve the energy-efficiency of compute and storage subsystems in data-centric computing. However, in a balanced system design, these changes call for matching improvement in the network subsystem as well. Conventional Ethernet-based networks are a potential energy-efficiency bottleneck due to the limited performance of gigabit Ethernet and the high power overhead of 10-gigbit Ethernet. In this paper, we evaluate the benefits of using an alternative, high-bandwidth, low-power, interconnect---PCIe---for power-efficient networking. Our experiments using PCIe's Non-Transparent Bridging for data transfer demonstrate significant performance gains at lower power, leading to 60--124% better energy efficiency. Early experiences with PCIe clustering also point to several challenges of PCIe-based networks and new opportunities for low-latency power-efficient datacenter networking.