Clustering Document based Semantic Similarity System using TFIDF and K-Mean

Abstract

The steady success of the Internet has led to an enormous rise in the volume of electronic text records. Sensitive tasks are increasingly being used to organize these materials in meaningful bundles. The standard clustering approach of documents was focused on statistical characteristics and clustering using the syntactic rather than semantic notion. This paper provides a new way to group documents based on textual similarities. Text synopses are found, identified, and stopped using the NLTK dictionary from Wikipedia and IMDB datasets. The next step is to build a vector space with TFIDF and cluster it using an algorithm K-mean. The results were obtained based on three proposed scenarios: 1) no treatment. 2) preprocessing without derivation, and 3) Derivative processing. The results showed that good similarity ratios were obtained for the internal evaluation when using (txt-sentoken data set) for all K values. In contrast, the best ratio was obtained with K = 20. In addition, as an external evaluation, purity measures were obtained and presented V measure of (txt). -sentoken) and the accuracy scale of (nltk-Reuter) gave the best results in three scenarios for K = 20 as subjective evaluation, the maximum time consumed with the first scenario (no preprocessing), and the minimum time recorded with the second scenario (excluding derivation).