Document Retrieval Using Deep Learning

Abstract

Document Retrieval has seen significant advancements in the last few decades. Latest developments in Natural Language Processing have made it possible to incorporate context and complex lexical patterns to document representations. This opens new possibilities for developing advanced retrieval systems. Traditional approaches for indexing documents suggest averaging word and sentence encoding to form fixed-length document embeddings. However, the common bag-of-word approach fails to incorporate the semantic context, which can be critical for understanding document-query relevancy. We address this by leveraging Bidirectional Encoder Representations from Transformers (BERT) to create semantically rich document embeddings. BERT compensates the limitations of the Term Frequency Inverse Document Frequency (TF-IDF) by incorporating contextual embeddings. In this paper, we propose an ensemble of BERT and TF-IDF for a document retrieval system, where TFIDF and BERT together score the documents against a query, to retrieve a final set of top K documents. We critically compare our model against the standard TF-IDF method and demonstrate a significant performance improvement on MS MARCO data (Microsoft-curated data of Bing queries).