A fast and transparent communication protocol for co-resident virtual machines

Network I/O workloads are dominating in most of the Cloud data centers today. One way to improve inter-VM communication efficiency is to support co-resident VM communication using a faster communication protocol than the traditional TCP/IP commonly used for inter-VM communications regardless whether VMs are located on the same physical host or different physical hosts. Although several co-resident VM communication mechanisms are proposed to reduce the unnecessary long path through the TCP/IP network stack, to avoid communication via Dom0, and to reduce invocation of multiple hypercalls when co-resident inter-VM communication is concerned. Most state of the art shared memory based approaches focus on performance, with programming transparency and live migration support considered. However, few of them provides performance, live migration support, user-kernel-hypervisor transparency at the same time. In this paper, we argue that all three above aspects are fundamental requirements for providing fast and highly transparent co-resident VM communication. We classify existing methods into three categories by their implementation layer in software stack: 1) user libraries and system calls layer, 2) below socket layer and above transport layer, 3) below IP layer. We argue that the choice of implementation layer has significant impact on both transparency and performance, even for live migration support. We present our design and implementation of XenVMC, a fast and transparent residency-aware inter-VM communication protocol with VM live migration support. XenVMC is implemented in layer 2. It supports live migration via automatic co-resident VM detection and transparent system call interception mechanisms, with multilevel transparency guaranteed. Our initial experimental evaluation shows that compared with virtualized TCP/IP method, XenVMC improves co-resident VM communication throughput by up to a factor of 9 and reduces corresponding latency by up to a factor of 6.