Can MPI Benefit Hadoop and MapReduce Applications?

The Message Passing Interface (MPI) standard and its implementations (such as MPICH and OpenMPI) have been widely used in the high-performance computing area to provide an efficient communication infrastructure. This paper investigates whether MPI can be adapted to the data intensive computing area to substantially speed up Hadoop and MapReduce applications, by reducing communication overheads. Three specific issues are studied. First, is the potential for reducing communication overheads significant, if MPI is used? Second, what are the main technical challenges to adapt MPI to Hadoop? Third, what are the minimal extensions to the MPI standard that can help alleviate the challenges while promise to significantly improve performance? To answer the first question, we identify important and basic communication primitives in both MPI and Hadoop, and make fair comparisons of their performance through experiments. The results show that the potential for improvement could be high. To answer the second and the third questions, we analyze the Hadoop code base to identify communication related programmers' needs. Furthermore, we propose a minimal interface extension to the MPI standard (only one pair of library calls are added), which capture the key-value pair nature commonly found in data intensive computing. This extension is implemented in a prototype library called MPI-D. Benchmark tests based on simulation show that Hadoop augmented with MPI-D could significantly speed up MapReduce application performance.