Efficient one-copy MPI shared memory communication in Virtual Machines

Abstract

Efficient intra-node shared memory communication is important for high performance computing (HPC), especially with the emergence of multi-core architectures. As clusters continue to grow in size and complexity, the use of virtual machine (VM) technologies has been suggested to ease the increasing number of management issues. As demonstrated by earlier research, shared memory communication must be optimized for VMs to attain the native-level performance required by HPC centers. In this paper, we enhance intra-node shared memory communication for VM environments. We propose a one-copy approach. Instead of following the traditional approach used in most MPI implementations, copying data in and out of a pre-allocated shared memory region, our approach dynamically maps user buffers between VMs, allowing data to be directly copied to its destination. We also propose a grant/mapping cache to reduce expensive buffer mapping cost in VM environment. We integrate this approach into MVAPICH2, our implementation of MPI-2 library. For intra-node communication, we are able to reduce the large message latency in VM-based environments by up to 35%, and increase bandwidth by up to 38% even as compared with unmodified MVAPICH2 running in a native environment. Evaluation with the NAS Parallel Benchmarks suite shows up to 15% improvement.