Self-adaptive ontology-based focused crawling: A literature survey

Abstract

Web crawlers are known to us since the birth of the internet since 1990, as the web pages are interconnected among themselves and form a unique path along which the crawler travels to fetch the information requested by the user/author. But the traditional crawlers are not able to distinguish between the relevant and the partially relevant web pages. Due to this the crawler had to fetch a huge amount of data from the web even if the web was not fully relevant to the user. This resulted in formation of the crawlers that were committed to the single topic given by the user. These crawlers were known as focused crawlers. These focused crawlers do not crawl the whole web as opposed to the traditional crawlers, as they only crawl the specific part of the web that is related to the given topic. This paper summarizes different qualities of various focused crawlers at present. Basically it divides the focused crawler into two different classes namely Semantic and Social Semantic. Semantic Focused Crawlers uses the ontology to its advantage and to obtain the topics that are contextually related to the given topic. Social Semantic Focused Crawlers takes the advantages of the social networking sites to obtain the web pages that are contextually related to the given topic, and usually the pages are shared by the people that have some interest in some topic related to the queried topic.