Effect of multi-word features on the hierarchical clustering of web documents

Abstract

Contemporary search engines and other automated web tools are faced with the task of extracting relevant information from huge web archives. This is supposed to be a difficult task due to the semi-structured and unstructured nature of the web documents. Users need automated ways of organizing and cataloging the web documents so that they can be queried efficiently. Clustering is typically employed to organize web archives and to subsequently handle user queries. This paper analyzes the effect of including multi-word features on the performance of a hierarchical clustering algorithm. Noun sequences are the predominant features considered in our work, while most of the previous research uses n-grams as features. The paper also analyzes the effect of combining link and content based representations for the web documents and their inter-relationships on the clustering performance. Empirical evaluation of the hierarchical clustering engine suggests that including multi-word features enhances the performance of the hierarchical clustering algorithm with respect to precision.