Modified TF-IDF Term Weighting Strategies for Text Categorization

Abstract

Text mining is a well-known technique in the domain of information retrieval which derives high quality of information from the text. To develop strategies for such text processing, an appropriate domain representation is required. Vectorized Term Frequency and Inverse Document Frequency (TF-IDF) representation of documents is one of the current strategies in use. Traditional TF-IDF uses term frequencies and document frequencies to generate a weighted term which is used for document representation. This method works sufficiently well, however, it is quite simplistic and overlooks many details that should ideally be relevant while processing the text such as document length, frequency distribution etc. To handle those shortcomings, this paper proposes four vector representation of documents which is the modified version of the traditional TF-IDF. In order to check the performance of the proposed techniques, different state-of-the-art classifiers are used to classify a corpus of documents. Experimental results on different benchmark datasets show that the performances of different classifiers using the proposed techniques are better than the traditional TF-IDF.