Michelangelo Diligenti, Marco Maggini, Filippo Maria Pucci, F. Scarselli
{"title":"具有有限带宽的爬虫的设计","authors":"Michelangelo Diligenti, Marco Maggini, Filippo Maria Pucci, F. Scarselli","doi":"10.1145/1013367.1013441","DOIUrl":null,"url":null,"abstract":"This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads. The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth. The experimental results show the effectiveness of the proposed technique.","PeriodicalId":409891,"journal":{"name":"WWW Alt. '04","volume":"18 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2004-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"8","resultStr":"{\"title\":\"Design of a crawler with bounded bandwidth\",\"authors\":\"Michelangelo Diligenti, Marco Maggini, Filippo Maria Pucci, F. Scarselli\",\"doi\":\"10.1145/1013367.1013441\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads. The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth. The experimental results show the effectiveness of the proposed technique.\",\"PeriodicalId\":409891,\"journal\":{\"name\":\"WWW Alt. '04\",\"volume\":\"18 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2004-05-19\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"8\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"WWW Alt. '04\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/1013367.1013441\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"WWW Alt. '04","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/1013367.1013441","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads. The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth. The experimental results show the effectiveness of the proposed technique.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
