Two-level dynamic index pruning

Abstract

In this paper, we propose two-level dynamic index pruning for improving retrieval efficiency without degrading the quality of query results. Analyzing the ClueWeb09 data set, we observe that most terms appear in thousands of different websites, while internet search engines typically just display the top-10 search results. We conclude that retrieval efficiency would be substantially improved, if one could prune entire websites by knowing that the scores of all their web pages will not make it in the top-10 scores of the query. Thus, two-level dynamic index pruning utilizes a hierarchical document numbering scheme to subdivide posting lists into sorted runs of the pages of one website rather than the flat inverted index of all web pages. Experimental results on the ClueWeb09 data set illustrate the benefits of two-level dynamic index pruning for improving retrieval efficiency.