Implementation of MapReduce over structured peer-to-peer overlay of underutilized resources

Abstract

The growth of data at an unprecedented rate poses challenge on its storage and analysis. To cope up with this, new machines are procured on a regular basis. On the other hand, a lot of computing resources in Institute labs, government offices etc. remain underutilized, since they are used for Internet and basic utilities. The aggregation of these underutilized resources can be more than handful to satisfy the computing needs of any large scale computation. This paper presents an approach of applying MapReduce framework on non-dedicated nodes. The target of this work is to efficiently use underutilized resources such as storage, processing power etc. on nodes to perform MapReduce applications. We have implemented a prototype of our model by forming a peer-to-peer (P2P) relationship among them and compared the performance with Hadoop, which is a popular large scale data processing framework under similar churn conditions.