An online clustering algorithm for Chinese web snippets based on Generalized Suffix Array

Abstract

As the information on the Internet increases dramatically, the web search engine has become an indispensable tool to search and locate the required information. Web snippets clustering can classify the search results and help users to narrow the search scope. This paper presents an online clustering algorithm for Chinese web snippets using common substrings. The algorithm firstly preprocesses the results of a search engine and extracts common substrings using Generalized Suffix Array. Then it builds a snippet-snippet similarity matrix by calculating similarities between every two snippets using common substring-based dimensional model. At last, the algorithm groups the web snippets using an improved hierarchical clustering algorithm. Theoretical analysis and experiments show that compared to traditional Chinese web snippet clustering algorithms based on Chinese word segmentation, our algorithm performs better both in the efficiency of clustering and the readability of the generated cluster labels.