Semantic Document Clustering using K-means algorithm and Ward's Method

Abstract

Nowadays in the age of technology, textual documents are rapidly growing over the internet. Offline and online documents, websites, e-mails, social network and blog posts, are archived in electronic structured databases. It is very hard to maintain and reach these documents without acceptable ranking and provide demand clustering while there is classification without any details. This paper presents an approach based on semantic similarity for clustering documents using the NLTK dictionary. The procedure is done by defining synopses from IMDB and Wikipedia datasets, tokenizing and stemming them. Next, a vector space is constructed using TFIDF, and the clustering is done using the ward's method and K-mean algorithm. WordNet is also used to semantically cluster documents. The results are visualized and presented as an interactive website describing the relationship between all clusters. For each algorithm three scenarios are considered for the implementations: 1) without preprocessing, 2) preprocessing without stemming, and 3) preprocessing with stemming. The Silhouette metric and seven other metrics are used to measure the similarity with the five different datasets. Using the K-means algorithm, the best similarity ratio acquired from the Silhouette metric with (nltk-Reuters) dataset for all clusters, and the highest ratio is when k=10. Similarly, with Ward's algorithm, the highest similarity ratio of the Silhouette metric obtained using (IMDB and Wiki top 100 movies, and nltk-brown) datasets together for all clusters, and best similarity ratio is obtained when k=5 using the (IMDB and Wiki top 100 movies) dataset. The results are compared with the literature, and the outcome exposed that the Ward's method outperforms the results of K-means for small datasets.