In Vacuo and In Situ Evaluation of SIMD Codecs

Abstract

The size of a search engine index and the time to search are inextricably related through the compression codec. This investigation examines this tradeoff using several relatively unexplored SIMD-based codecs including QMX, TurboPackV, and TurboPFor. It uses (the non-SIMD) OPTPFor as a baseline. Four new variants of QMX are introduced and also compared. Those variants include optimizations for space and for time. Experiments were conducted on the TREC .gov2 collection using topics 701-850, in crawl order and in URL order. The results suggest that there is very little difference between these codecs, but that the reference implementation of QMX performs well.