XRT: Programming-Language Independent MapReduce on Shared-Memory Systems

Abstract

Increasing processor core-counts have created an opportunity for efficient parallel processing of large datasets on shared-memory systems. When compared to clusters of networked commodity hardware, shared-memory systems have the potential to provide better per-core performance, a more straightforward development environment and reduced operational overhead. This paper presents XRT, a high-performance and programming-language independent MapReduce runtime for shared-memory systems. XRT is built to be simple to use, pedantic about resource usage and capable of utilizing disk-based data structures for processing datasets too large to fit in memory. To our knowledge, XRT is the first MapReduce runtime explicitly designed for programming-language independent MapReduce. Moreover, XRT is the first MapReduce runtime for shared-memory systems taking advantage of disk-based data structures for processing datasets which cannot fit in memory. Benchmarks of three common data processing problems demonstrate the disk-based capabilities as well as the excellent speedup profile of XRT as system core-counts increase.