Improving document representation using KPCA and clustered word embeddings

Abstract

Text mining approaches have shown to be an efficient way to examine publicly available online information on a variety of topics. However, because of the lack of context, online texts are difficult to represent. Word embedding techniques, such as word2vec, capture semantic relationships between words particularly well when trained on large text collections; but they fail when trained on small datasets. This paper proposes a document representation method based on word clusters. In this approach, Word embeddings created by word2vec are supplemented with morphological information collected from kernel principal component analysis (KPCA) computed on word similarity. After running KPCA, a spherical k-means algorithm is applied, with centroid representing the topic of that cluster. Next, the document is vectorized using the frequencies of these clusters. The proposed approach successfully includes the effects of semantically and morphologically related terms on document proximity through these data-driven notions. When combined with an appropriate weighting system, the proposed approach enables improved document representation and interpretability of the resulted document vectors.