A character elimination algorithm for lossless data compression

Abstract

Summary form only given. We present a detailed description of a lossless compression algorithm intended for use on files with non-uniform character distributions. This algorithm takes advantage of the relatively small distances between character occurrences once we remove the less frequent characters. This allows it to create a compressed version of the file that, when decompressed, is an exact copy of the file that was compressed. We begin by performing a Burrows-Wheeler (1994) Transform (BWT) on the file. The algorithm scans this BWT file to create a character frequency model for the compression phase. To deal with the issue of bit encoding, we write every number as a byte or sequence of bytes to the compressed file and run an arithmetic encoder after the file has been compiled.