A TWO-STAGE KEYWORD BASED CRAWLER FOR GATHERING DEEP-WEB SITES

Abstract

Deep web is termed as sites present on web but not accessible by any search engine. Due to the large volume of web resources and the dynamic nature of deep web, achieving wide coverage and high efficiency is a challenging issue. Keyword based crawler for hidden web interfaces consist of mainly two stages, first is site locating another is in-site exploring. Site locating starts from seed sites and obtains relevant websites through reverse searching and obtains relevant sites through feature space of URL, anchor and text around URL. Second stage receives input from site locating and starts to find relevant link from those sites. The adaptive link learner is used to find out relevant links with help of link priority and link rank. To eliminate inclination on visiting some more closely related links in inaccessible web directories, we design a data structure called link tree to achieve broader coverage for a website.