A Parameter-Adjustable Estimating Method for Change Frequency of Web Pages

Abstract

Refreshing on-line information in time is a main task of incremental crawling, so it is very important to predict the change frequency of web pages. We model the change of page as a Poisson process in this paper. And based on it, we propose a parameter-adjustable algorithm after considering the unbiasedness, efficiency and consistency comprehensively. This algorithm can adjust the parameters in order to estimate the change frequency more effective.