网站普查

2009 International Conference for Internet Technology and Secured Transactions, (ICITST) Pub Date : 2009-11-01 DOI:10.1109/ICITST.2009.5402623

A. Qadeer, Waqar Mahmood, A. Waheed

{"title":"网站普查","authors":"A. Qadeer, Waqar Mahmood, A. Waheed","doi":"10.1109/ICITST.2009.5402623","DOIUrl":null,"url":null,"abstract":"The website census is an effort to enumerate all the websites on the World Wide Web (WWW) without using crawling. Crawling is a traditional way of website discovery. It is conceptually simple but the very size of the WWW makes the implementation complex and resource demanding. The enormous amount of bandwidth, a huge persistent storage pool, a sufficiently large cluster of machines for data processing and a complex set of software systems are just a few examples of the needed resources. In this work, we use exhaustive IP range probing to detect the presence of a web server on TCP port 80. Although this probing is exhaustive in nature, it is lightweight in terms of resource demands. This enumeration of websites has many applications. The most obvious is to use it as a seed to the conventional crawling. It can be refined to be used as a top level domain (TLD) specific seed for targeted crawling.","PeriodicalId":251169,"journal":{"name":"2009 International Conference for Internet Technology and Secured Transactions, (ICITST)","volume":"47 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2009-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":"{\"title\":\"The website census\",\"authors\":\"A. Qadeer, Waqar Mahmood, A. Waheed\",\"doi\":\"10.1109/ICITST.2009.5402623\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The website census is an effort to enumerate all the websites on the World Wide Web (WWW) without using crawling. Crawling is a traditional way of website discovery. It is conceptually simple but the very size of the WWW makes the implementation complex and resource demanding. The enormous amount of bandwidth, a huge persistent storage pool, a sufficiently large cluster of machines for data processing and a complex set of software systems are just a few examples of the needed resources. In this work, we use exhaustive IP range probing to detect the presence of a web server on TCP port 80. Although this probing is exhaustive in nature, it is lightweight in terms of resource demands. This enumeration of websites has many applications. The most obvious is to use it as a seed to the conventional crawling. It can be refined to be used as a top level domain (TLD) specific seed for targeted crawling.\",\"PeriodicalId\":251169,\"journal\":{\"name\":\"2009 International Conference for Internet Technology and Secured Transactions, (ICITST)\",\"volume\":\"47 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2009-11-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2009 International Conference for Internet Technology and Secured Transactions, (ICITST)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/ICITST.2009.5402623\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2009 International Conference for Internet Technology and Secured Transactions, (ICITST)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICITST.2009.5402623","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 1

摘要

网站普查是在不使用爬行的情况下枚举万维网(WWW)上的所有网站。爬行是一种传统的网站发现方式。它在概念上很简单，但是WWW的大小使得实现变得复杂并且需要大量的资源。巨大的带宽、巨大的持久存储池、用于数据处理的足够大的机器集群和一组复杂的软件系统只是所需资源的几个例子。在这项工作中，我们使用详尽的IP范围探测来检测TCP端口80上web服务器的存在。虽然这种探查本质上是详尽的，但就资源需求而言，它是轻量级的。这个网站列表有很多应用。最明显的是用它作为常规爬行的种子。它可以被细化为用于目标爬行的顶级域(TLD)特定种子。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

The website census

The website census is an effort to enumerate all the websites on the World Wide Web (WWW) without using crawling. Crawling is a traditional way of website discovery. It is conceptually simple but the very size of the WWW makes the implementation complex and resource demanding. The enormous amount of bandwidth, a huge persistent storage pool, a sufficiently large cluster of machines for data processing and a complex set of software systems are just a few examples of the needed resources. In this work, we use exhaustive IP range probing to detect the presence of a web server on TCP port 80. Although this probing is exhaustive in nature, it is lightweight in terms of resource demands. This enumeration of websites has many applications. The most obvious is to use it as a seed to the conventional crawling. It can be refined to be used as a top level domain (TLD) specific seed for targeted crawling.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

2009 International Conference for Internet Technology and Secured Transactions, (ICITST)

自引率

0.00%

发文量