IMPLEMENTATION OF SMARTCRAWLER FOR DEEP-WEB SITES

Abstract

Deep web is termed as data present on web but inaccessible to search engine. Due to the large volume of web resources and the dynamic nature of deep web, achieving wide coverage and high efficiency is a challenging issue. Smart crawler for hidden web interfaces consist of mainly two stages, first is site locating another is in-site exploring. Site locating starts from seed sites and obtains relevant websites through reverse searching and obtains relevant sites through feature space of URL, anchor and text around URL. Second stage takes input from site locating and goes to find relevant link from those sites. The adaptive link learner is used to find out relevant links with help of link priority and link rank.. To eliminate bias on visiting some highly relevant links in hidden web directories, we design a link tree data structure to achieve wider coverage for a website.