带有URL签名的网络爬虫-性能研究

2012 4th Conference on Data Mining and Optimization (DMO) Pub Date : 2012-10-15 DOI:10.1109/DMO.2012.6329810

Lay-Ki Soon, Yee-Ern Ku, Sang Ho Lee

{"title":"带有URL签名的网络爬虫-性能研究","authors":"Lay-Ki Soon, Yee-Ern Ku, Sang Ho Lee","doi":"10.1109/DMO.2012.6329810","DOIUrl":null,"url":null,"abstract":"URL signature was proposed to be implemented in web crawling, aiming to avoid processing duplicated web pages for further web crawling. In this paper, we present our performance study on an open source web crawler - WebSPHINX, in which we have embedded URL signature. The experimental result indicates that URL signature is able to reduce the processing of duplicated web pages significantly for further web crawling at a negligible cost compared to the one without URL signature.","PeriodicalId":330241,"journal":{"name":"2012 4th Conference on Data Mining and Optimization (DMO)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2012-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"5","resultStr":"{\"title\":\"Web crawler with URL signature — A performance study\",\"authors\":\"Lay-Ki Soon, Yee-Ern Ku, Sang Ho Lee\",\"doi\":\"10.1109/DMO.2012.6329810\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"URL signature was proposed to be implemented in web crawling, aiming to avoid processing duplicated web pages for further web crawling. In this paper, we present our performance study on an open source web crawler - WebSPHINX, in which we have embedded URL signature. The experimental result indicates that URL signature is able to reduce the processing of duplicated web pages significantly for further web crawling at a negligible cost compared to the one without URL signature.\",\"PeriodicalId\":330241,\"journal\":{\"name\":\"2012 4th Conference on Data Mining and Optimization (DMO)\",\"volume\":\"1 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2012-10-15\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"5\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2012 4th Conference on Data Mining and Optimization (DMO)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/DMO.2012.6329810\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2012 4th Conference on Data Mining and Optimization (DMO)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/DMO.2012.6329810","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 5

摘要

提出了在网页抓取中实现URL签名的方法，以避免处理重复的网页进行进一步的网页抓取。在本文中，我们展示了我们对一个开源网络爬虫WebSPHINX的性能研究，我们在其中嵌入了URL签名。实验结果表明，与不使用URL签名的方法相比，URL签名可以显著减少重复网页的处理，而成本可以忽略不计。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Web crawler with URL signature — A performance study

URL signature was proposed to be implemented in web crawling, aiming to avoid processing duplicated web pages for further web crawling. In this paper, we present our performance study on an open source web crawler - WebSPHINX, in which we have embedded URL signature. The experimental result indicates that URL signature is able to reduce the processing of duplicated web pages significantly for further web crawling at a negligible cost compared to the one without URL signature.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

2012 4th Conference on Data Mining and Optimization (DMO)

自引率

0.00%

发文量