Improve precategorized collection retrieval by using supervised term weighting schemes

Abstract

The emergence of the World Wide Web has led to an increased interest in methods for searching for information. A key characteristic of many online document collections is that the documents have pre-defined category information, such as the variety of scientific articles accessible via digital libraries (e.g. ACM, IEEE, etc.), medical articles, news-wires and various directories (e.g. Yahoo, OpenDirectory Project, etc.). However, most previous information retrieval systems have not taken the pre-existing category information into account. In this paper, we present weight adjustment schemes based upon the category information in the vector-space model, which are able to select the most content-specific and discriminating features. Our experimental results on TREC data sets show that the pre-existing category information does provide additional beneficial information to improve retrieval. The proposed weight adjustment schemes perform better than the vector-space model with the inverse document frequency (IDF) weighting scheme when queries are less specific. The proposed weighting schemes can also benefit retrieval when clusters are used as an approximations to categories.