Application and research of heuristic search algorithm in crawler field

Abstract

Breadth first search algorithm is widely used in traditional crawler programs. This kind of search algorithm takes the way of blindly searching data, and does not make use of the characteristics of crawling data. When there are many websites, the crawler will be very time-consuming. Based on the above reasons, an adaptive heuristic search algorithm is designed. According to the relevance between the page and the topic and the related information of the link, the more likely link is selected for crawling, and the estimation function is modified according to the crawling result. Experiments show that the algorithm can significantly improve the efficiency of crawler than breadth first search algorithm.