Design And Implementation of Web Crawler Based on 'Internet +'Data Automatic Extraction

Abstract

The rise of the strategy of “Internet +” breaks the barriers of data and information. Web crawler is widely used in data acquisition and data analysis in the massive Internet plus information. Taking “IMDB top250 movies” as the goal, using the crawler technology based on Python language, this paper explains the four steps of web crawler in detail, compares the differences of three web page parsing methods: BeautifulSoup, Regular Expression(Re) and XPath, and completes the crawling of target data. The experimental results show that Re is the best in data analysis speed; In terms of web page parsing logic, beautiful soup is the best; From the perspective of comprehensive use, XPath is more suitable.