An efficient Internet crawling and filtering system for the nationwide tendering information retrieval

Abstract

With the growth of Internet, the central government and local governments have begun to publish matters concerning the prospect of orders for public works, the announcement of tendering and the contracting information on their Web sites. However, it is time consuming and painful for bidders such as constructors and manufacturers to periodically search the above information that matches their needs. Recently, there are various search engines, e.g. Google and Yahoo!, but those general search engines are not effective for the purpose of retrieving the above information quickly enough because of their crawling interval and coverage. Then we developed a system to automate the process of gathering such information, filtering for users' needs and delivering as the tendering and contracting information database. We describe the concept of the system as well as the key techniques to realize it: (1) to efficiently retrieve only relevant Web pages, and (2) filtering to match users' needs.