An Automatic Label Extraction Technique for Domain-Specific Hidden Web Crawling (LEHW)

Abstract

General-purpose search engines (e.g. Google and Yahoo) ignore valuable data that represent 80% of the content on the Web, this portion of Web called hidden Web (HW). Pages in the hidden Web are dynamically generated in response to queries submitted via the search forms. In this paper, a new algorithm for extracting labels from multi-attribute (M-A) search form fields is proposed. A technique for automatic query generation for single-attribute (S-A) search forms is also provided in order to enhance the performance of the overall domain-specific hidden Web crawlers. The innovation of (LEHW) algorithm is its capability to distinguish between (S-A) and (M-A) forms; so that the capability of dealing with both of them, unlike most hidden Web crawlers that ignore either of them. Embedding of the proposed algorithm within the overall framework of the HW crawler is evaluated through experiments using real Web sites. The preliminary results demonstrate the accuracy and precision of the proposed approach for most of the sites considered