ThinDedup: An I/O Deduplication Scheme that Minimizes Efficiency Loss due to Metadata Writes

I/O deduplication is an important technique for saving I/O bandwidth and storage space for storage systems. However, it requires a new level of address mapping, and consequently needs to maintain corresponding metadata. To meet requirements on data persistency and consistency, the metadata writing is likely to make deduplication operations much fatter, in terms of amount of additional writes on the critical I/O path, than one might expect. In this paper we propose to compress the data and insert metadata into data blocks to reduce metadata writes. Assuming that performance-critical data are usually compressible, we can mostly remove separate writes of metadata out of the critical path of servicing users' requests, and make I/O deduplication much thinner. Accordingly we name the scheme ThinDedup. In addition to metadata insertion, ThinDedup also uses persistency of data fingerprints to evade enforcement of write order between data and metadata. We have implemented ThinDedup in the Linux kernel as a device mapper target to provide block-level deduplication. Experimental results show, compared to existing deduplication schemes, ThinDedup achieves (much) higher (up to 3X) I/O throughput and lower latency (reduced by up to 88%) without compromising data persistency.