A novel term weighting scheme with distributional coefficient for text categorization with support vector machine

Abstract

In text categorization, vectorizing a document by probability distribution is an effective dimension reduction way to save training time. However, the data sets that share many common keywords between categories affect the classification performance seriously. To address that problem, firstly, we conduct an effective term weighting scheme consisting of posterior probability and relevance frequency to improve the performance of the traditional hybrid classification model. To get a better representation of the information contained in a document, as well as the introduction of an advanced hybrid classification model, we also propose a novel term weighting scheme with distributional coefficient so as to obtain further accuracy enhancement. The experimental results show that these proposed schemes are significantly better than the traditional method.