Design And Implementation of Web Crawler Based on 'Internet +'Data Automatic Extraction

Lulu Zhang, Junru Li, Dacheng Feng, Junjie Sun
{"title":"Design And Implementation of Web Crawler Based on 'Internet +'Data Automatic Extraction","authors":"Lulu Zhang, Junru Li, Dacheng Feng, Junjie Sun","doi":"10.1109/ICCECE58074.2023.10135210","DOIUrl":null,"url":null,"abstract":"The rise of the strategy of “Internet +” breaks the barriers of data and information. Web crawler is widely used in data acquisition and data analysis in the massive Internet plus information. Taking “IMDB top250 movies” as the goal, using the crawler technology based on Python language, this paper explains the four steps of web crawler in detail, compares the differences of three web page parsing methods: BeautifulSoup, Regular Expression(Re) and XPath, and completes the crawling of target data. The experimental results show that Re is the best in data analysis speed; In terms of web page parsing logic, beautiful soup is the best; From the perspective of comprehensive use, XPath is more suitable.","PeriodicalId":120030,"journal":{"name":"2023 3rd International Conference on Consumer Electronics and Computer Engineering (ICCECE)","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-01-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2023 3rd International Conference on Consumer Electronics and Computer Engineering (ICCECE)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICCECE58074.2023.10135210","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

The rise of the strategy of “Internet +” breaks the barriers of data and information. Web crawler is widely used in data acquisition and data analysis in the massive Internet plus information. Taking “IMDB top250 movies” as the goal, using the crawler technology based on Python language, this paper explains the four steps of web crawler in detail, compares the differences of three web page parsing methods: BeautifulSoup, Regular Expression(Re) and XPath, and completes the crawling of target data. The experimental results show that Re is the best in data analysis speed; In terms of web page parsing logic, beautiful soup is the best; From the perspective of comprehensive use, XPath is more suitable.
基于“互联网+”数据自动提取的网络爬虫的设计与实现
“互联网+”战略的兴起,打破了数据和信息的壁垒。在海量的互联网+信息中,网络爬虫被广泛应用于数据采集和数据分析。本文以“IMDB top250电影”为目标,利用基于Python语言的爬虫技术,详细阐述了网络爬虫的四个步骤,比较了BeautifulSoup、正则表达式(Re)和XPath三种网页解析方法的差异,完成了目标数据的爬虫。实验结果表明,Re在数据分析速度上是最好的;在网页解析逻辑方面,美汤是最好的;从综合使用的角度来看,XPath更合适。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信