New approach of smoothing to extend language model in Lucene

Abstract

This paper focuses on extending Information Retrieval (IR) API named Lucene by implementing various smoothing techniques. Lucene is a free API written entirely in JAVA. It allows to create an indexing and search engine for textual files. Our contribution has two parts: first, we propose a smoothing approach that we integrated into Lucene’s language model in order to improve its search abilities. The suggested approach is based on a combination of algorithms already implemented in Lucene. Next, we have implemented the Absolute Discount smoothing approach and integrated it into Lucene’s language model. Our proposed approaches have been evaluated in information retrieval on test collection. Our contribution yielded very good search results in some cases compared to other approaches implemented in Lucene.