Multithreaded Processing in Dynamic Inverted Indexes for Web Search Engines

Processing queries in Web search engines demands the efficient use of hardware resources to cope with the scale and dynamics of user traffic. This paper focuses on the multithreaded processing of queries that requires (1) accessing a large inverted index data structure to obtain a set of documents, (2) rank them by executing the WAND operator in order to obtain the top K most pertinent documents for the query, and (3) resolve the insertion of new documents on the inverted index concurrently with the execution of queries. We propose an efficient strategy to assign threads to queries and index update operations which is suitable to support updates on the index concurrently with query processing. The core of our proposal is a simple classification technique devised to quickly assign threads to query operations.