Notice of Violation of IEEE Publication PrinciplesPipelined-MapReduce: An Improved MapReduce Parallel Programing Model

Abstract

MapReduce is a parallel programming model, and used to handle large datasets. The MapReduce program can be automatically concurrent executed in large-scale commodity machines. We proposed an improved MapReduce programming model -- Pipelined-MapReduce, to solve the data intensive of information retrieval problems. Pipelined-MapReduce allows data transfer by pipeline between the operations, expanding the batched MapReduce programming model, and can reduce the completion time, and improve the system utilization rate. The experimental results demonstrate that the implemention of Pipelined-MapReduce can scale well and efficiently process large datasets on commodity machines.