Inverted Index Compression Using Multi-codes

Abstract

How to decrease the space consumed by index is a key issue in big data processing. In this paper, a new compression method is proposed to decrease the space consumption of inverted index. First, a lot of redundant integers are removed by using the techniques of splitting inverted list, adding tags and making groups. Second, the total number of small integers is increased by using d-gaps in each group. Third, these sub sequences are compressed using different codes. At last, all compressed sub sequences are combined into a long sequence. Experiment results show that our method decreases the compression ratio and its decoding speed is also fast.