Efficient Management of Semi-Persistent Data for the Evolving Web

Abstract

The Web is an information repository that grows and evolves fast. Traditional data management systems are based on a persistence model that are not suited for management of Web data. In this paper, we propose a semi-persistence model to capture the evolving nature of the Web. By semi-persistence, we mean data with relaxed persistence requirement where obsolete data may be moved to somewhere or removed implicitly and autonomously. In a semi-persistent data management system, data and the associated statistics have to be maintained efficiently to support trend-report queries and age estimation. We propose a space-efficient data structure, called moving bloom filters (MBF) to maintain time-sensitive statistics of underlying data. The preliminary experiments show that the optimized MBF achieves considerable improvement on space usage while maintaining the same precise estimation of frequency statistics.