Compression experiments on term-document index

Abstract

The increase in the size of the data used in natural language processing activities brings with it time and space constraints. Thus, it is important to both store and access data efficiently. This study includes experiments for storing the term-document index, which will be used in a natural language processing project, effectively in memory. For this purpose, the indexed data is compressed using Run-Length coding and then Huffman coding algorithm. Compression experiments have been conducted with new versions of Huffman Coding which are arranged in a structure suitable for indexing data used in the study. The results were compared with the results of commonly used compression tools and the results turned out to be successful.