Automatic information discovery from the "invisible Web"

Abstract

A large amount of online information resides on the "invisible Web" - Web pages that are generated dynamically from databases and other data sources hidden from the user. They are not indexed by a static URL but are generated when queries are made via a search interface (a specialized search engine). In this paper, we propose a system that is capable of automatically making use of these specialized engines to find information on the invisible Web. We describe our overall architecture and process: from obtaining the search engines to picking the right engines to query. Experiments show that we can find information that is not found by traditional search engines.