FICA: A Fast Intelligent Crawling Algorithm

Abstract

Due to the proliferation and highly dynamic nature of the Web, an efficient crawling and ranking algorithm for retrieving the most important pages has remained as a challenging issue. Several algorithms like PageRank (Page et al., 1998) and OPIC (Abiteboul et al., 2003) have been proposed. Unfortunately, they have high time complexity. In this paper, an intelligent crawling algorithm based on reinforcement learning, called FICA is proposed that models a real surfing user. The priority for crawling pages is based on a concept which we name as logarithmic distance. FICA is easy to implement and its time complexity is 0(E*logV) where V and E are the number of nodes and edges in the Web graph respectively. Comparison of the FICA with other proposed algorithms shows that FICA outperforms them in discovering highly important pages. Furthermore FICA computes the importance (ranking) of each page during the crawling process. Thus, we can also use FICA as a ranking method for computation of page importance. We have used UK's Web graph for our experiments.