An implementation of heterogeneous architecture based MapReduce in the clouds

Abstract

With the rapid development of computer technology, heterogeneous architecture based MapReduce (HA-MapReduce for short) is widely studied in the big data processing domain. Meanwhile, cloud computing is becoming an important alternative for providing computational infrastructure. Therefore, in this paper, we propose an implementation of HA-MapReduce in the cloud environment. First, we design a uniform MapReduce framework for heterogeneous architecture, which can utilize CPU and coprocessor cooperatively and efficiently. Second, we propose a coprocessor token mechanism for handling the coprocessor scalability and fault tolerance issues. Finally, we design a lightweight virtualization based cloud platform for low overhead and easy deployment. We deploy a CPU-MIC heterogeneous cluster for our HA-MapReduce and cloud platform. The experimental results show that our system is up to 1.21× and 1.38× faster than VM-based cloud platform, and 2.31× to 8.39× speedups than CPU-based Hadoop.