Building a robust and compact search index

Abstract

With exponential data growth search engines require more memory for storage and time for search. The data is indexed to increase search speed, which requires additional memory. In this study we develop a fully functional search engine for Wikipedia articles and compare different indexing techniques. Using vector quantization for compression we fit an index into a single machine’s RAM. Moreover, we show that by using metadata and additional search for the out-of-vocabulary words we improve the overall system’s quality.