Using Normalized Compression Distance for Classifying File Fragments

Abstract

We have applied the generalized and universal distance measure NCD--Normalized Compression Distance--to the problem of determining the types of file fragments via example. A corpus of files that can be redistributed to other researchers in the field was developed and the NCD algorithm using k-nearest-neighbor as a classification algorithm was applied to a random selection of file fragments. The experiment covered circa 2000 fragments from 17 different file types. While the overall accuracy of the n-valued classification only improved the prior probability of the class from approximately 6% to circa 50% overall, the classifier reached accuracies of 85%--100% for the most successful file types.