Credibility-based result verification for Map-reduce

Abstract

The Hadoop architecture and Map-reduce paradigm together provide a cost-effective distributed computing environment for large data banks. The system performs effective, fault-tolerant and speedy processing of data by replicating data and computation on multiple nodes. This can lead to a scenario known as collusion where malicious nodes might join hands and return wrong results. Thus, there is a need to verify the results. Majority voting scheme is the most common approach, but it suffers from the problem of declaring the results returned by malicious nodes as acceptable if those nodes form the majority. Thus, majority alone cannot be used to ascertain the correctness of a result. We propose a credibility-based approach for result verification. It assigns credibility values to nodes based on their execution outcomes and uses them effectively for result verification. Our experiments show that the proposed approach is much more accurate than majority-based scheme.