Duplications and Misattributions of File Fragment Hashes in Image and Compressed Files

Abstract

Hashing is used in a wide variety of security contexts. Hashes of parts of files, fragment hashes, can be used to detect remains of deleted files in cluster slack, to detect illicit files being sent over a network, to perform approximate file matching, or to quickly scan large storage devices using sector sampling. In this work we examine the fragment hash uniqueness and hash duplication characteristics of five different data sets with a focus on JPEG images and compressed file archives. We consider both block and rolling hashes and evaluate sizes of the hashed fragments ranging from 16 to 4096 bytes. During an initial hash generation phase hash metadata is created for each data set, which in total becomes several several billion hashes. During the scan phase each other data set is scanned and hashes checked for potential matches in the hash metadata. Three aspects of fragment hashes are examined: 1) the rate of duplicate hashes within each data set, 2) the rate of hash misattribution where a fragment hash from the scanned data set matches a fragment in the hash metadata although the actual file is not present in the scan set, 3) to what extent it is possible to detect fragments from files in a hashed set when those files have been compressed and embedded in a zip archive. The results obtained are useful as input to dimensioning and evaluation procedures for several application areas of fragment hashing.