An Intelligent Topic Web Crawler Based on DTB

Abstract

Web crawling is a fundamental step in many Web applications, such as search engine and data mining. This paper proposes an intelligent topic Web crawler based on DTB (dynamic topic base), which through studying on Web crawlers which filter URLs based on different methods. This Web crawler can update the topic base automatically and improve the accuracy of URL filtering. Experimental results show that the proposed Web crawler can fetch more topic relevant Web pages by crawling less Web space and in less time.