Extraction and normalization of IR indexing terms and phrases in a highly inflectional language

Abstract

Term-based indexing of documents is conventionally implemented by stemmers or their corpus-based improvements, both of which encode implicit linguistic information. Terms are directly derived from document content such that a unique indexing approach is available at indexing run-time. For highly inflectional languages where term variation is high, such techniques are more error-prone. The main focus of the current study is the extraction and normalization of single terms and phrases and the proposal of authenticated control of indexing. The proposed approach relies on the use of explicit linguistic knowledge, appropriately encoded in large language resources. Such control guarantees the highest possible expansion factor for indexing terms as well as indexing consistency. Moreover, it offers a framework where different and eventually contradicting indexing criteria can be practiced, conventional and Natural Language Processing (NLP)-based Information Retrieval (IR) applications can be served, while adaptations can be made for tuning to a specific domain or corpus.