RMD: A Resemblance and Mergence Based Approach for High Performance Deduplication

Data deduplication, a data redundancy elimination technique, has been employed in almost all kinds of application environments to reduce storage space. However, one of the main challenges facing deduplication technology is to provide a fast key-value fingerprint index for large datasets, as the index performance is critical to the overall deduplication performance. This paper proposes RMD, a resemblance and mergence based deduplication scheme, which aims to provide quick responses to fingerprint queries. The key idea of RMD is to leverage a bloom filter array and the data resemblance algorithm to dramatically reduce the query range for deduplication. Moreover, RMD utilizes mergence based approach to merge resemblance segments to relevant bins, and exploits frequency-based Fingerprint Retention Policy to reduce the bin capacity to improve query throughput and improve data deduplication ratio. Extensive experimental results with real-world datasets have shown that RMD is able to achieve pretty high query performance and outperforms several state-of-the-art deduplication schemes.