SharpSpider: spidering the Web through Web services

Abstract

Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index buiU after successive crawls of the Web.