A New Conceptual Approach to Document Indexing

This paper presents a new conceptual indexing technique intended to overcome the major problems resulting from the use of Term Frequency (TF) based approaches. To resolve the semantic problems related to TF approaches, the proposed technique disambiguates the words contained in a document and creates a list of super ordinates based on an external knowledge source. In order to reduce the dimension of the document vector, the final set of index values is created by extracting a set of common concepts, shared by multiple related words, from the list of hypernyms. Subsequently, a weight is assigned to each concept index by considering its position in the knowledge source's hierarchical tree (i.e. distance from the substituted words) and its number of occurrences. By applying the proposed technique, we were able to disambiguate words within different contexts, extrapolate concepts from documents, assigning appropriate normalised weights, and significantly reduce the vector dimension.