san_sim: Factual and efficient URL text similarity algorithm

Abstract

Similarity determines the relation between two objects. We need this to establish an order between the two objects being compared. Here we want to compare two URLs (Uniform Resource Locater) and find which is more relevant to the input query. Content mining is one of web mining technique which uses text of the web page. Online learning is used where entire dataset cannot be used at training time because of its size. Here few popular text similarity methods are implemented and their relevance is compared with our proposed method. We find that our algo-rithmperforms better than the traditional tex similarity measures such as LCS (Longest Common Sequence) and Dice score. Performance of our proposed method is better as higher Precision, Recall and F measures are achieved. This proves that data specific filtering methods, online learning principles when used with statistical method produces better result.