A framework for utilising usage trends in the crawling and indexing process of search engines

Abstract

Making search engines responsive to human needs requires understanding of user navigations through the search results in response to the submitted queries. The user behaviour characterisation provides an interesting perspective towards understanding the workload imposed on the search engine and can be used to address crucial points such as load balancing, content caching, data distribution and result optimisation. The user browsing behaviour is recorded in the query logs of search engines and usually referred to as web usage data. In this paper, a technique to utilise the users' browsing behaviour at the crawling and indexing process is being proposed so as to direct the crawler to download the important pages, which were not previously crawled. As the work attempts to index most of important pages based on user feedback, it would benefit the search engine to enhance its efficiency. To add further to the proposed work, the existing data structures maintained by the search engines has been refined so as to support the proposed user feedback mechanism and open more research directions.