SAFSB: A self-adaptive focused crawler

Abstract

There are about 3 billion indexed websites present in the WWW. Not all websites do not belong to a particular topic are indexed by a search engine say google.com, there are online platforms available where different users help the person asking for a (Universal Resource Locator) URL containing a topical information. To verify the authenticity and validity of the URL, an empirical methodology and its ranking to major its relevancy is presented through this paper. To semantically expand the search, topic ontology is used for the pre-processing of the focused crawler to make search more effective. The performance of our web crawler is further increased by using the ontology based learning which is constantly being updated by dictionary based learning and related words of the named entities. The harvest ratio is used which represents the ratio between the relevant pages and the crawled pages shows a significant improvement than the previous methods.