Bucket based data deduplication technique for big data storage system

Abstract

In this paper proposed bucket based data deduplication technique is presented. In proposed technique bigdata stream is given to the fixed size chunking algorithm to create fixed size chunks. When the chunks are obtained then these chunks are given to the MD5 algorithm module to generate hash values for the chunks. After that MapReduce model is applied to find whether hash values are duplicate or not. To detect the duplicate hash values MapReduce model compared these hash values with already stored hash values in bucket storage. If these hash values are already present in the bucket storage then these can be identified as duplicate. If the hash values are duplicated then do not store the data into the Hadoop Distributed File System (HDFS) else then store the data into the HDFS. The proposed technique is analyzed using real data set using Hadoop tool.