Achieving Performance and Programmability for MapReduce(-Like) Frameworks

Abstract

Programmability and performance are often considered alternatives in the context of HPC programming systems. For example, general purpose frameworks like MPI are associated with high performance, and though MapReduce and similar frameworks have demonstrated high programmability, it is also well accepted that they fall short in terms of performance. Providing abstractions that maintain high programmability and performance remains an open question. In this paper, we introduce two different variations of the original MapReduce API, We demonstrate efficient implementations of the three APIs, focusing on how the API differences impact middleware implementation, and examine the resulting performance. Furthermore, to understand how application characteristics impact relative performance of the three systems, we develop and validate a performance model. Overall, we show that a MapReduce-like AP that only requires small additional effort from programmers can provide high performance, outperforming Spark significantly.