Data-Intensive Services for Large-Scale Archive Access

Abstract

Recently many organizations have accumulated data from such various sources as web and network sensors and constructed large-scale archives. Some would like to publish their archives to public to facilitate the activities of other organizations, but the scale of the archives causes problems. Therefore, we propose the concept of data-intensive services, which publish large-scale archives. We show the architecture for data-intensive services and focus on the following fundamental functional properties: 1) enhancing search, 2) preprocessing, 3) and asynchronous transfer. We also developed a reference implementation of a framework for data-intensive services and applied it to a web archive that contains about 2 billion documents and greatly improved the access performance to the web archive at small development cost.