Measuring the Similarity of Files by Data Compression

Abstract

The two meta-algorithms Concat Compress and Cross Compress, which can be used to measure the similarity of files, were subjected to an extensive practical test together with the compression algorithms Re-Pair, gzip and bz2:Five labeled datasets with 6533 entries and approximately 10 MB were subjected to a classification procedure using these algorithms. Theoretical considerations of the two meta-algorithms have been made in the past [1], but the practical implementation of these methods is still in its infancy. The results from our experiments are promising and show the great potential of this approach.