Empirical long-term observation on the mobile Web update frequency

Abstract

The mobile Internet extends its coverage and opens new research and business opportunities. For example, mobile Internet search engines are still unexplored. In search engine design, it is important to capture the dynamic characteristics of the mobile Web. In order to identify such characteristics, the author designed MARCH-1, a mobile crawling system that regularly fetches and archives the mobile Web content. From the 2-year observation, the author attempts to identify the dynamism in the mobile Web. The author analyzes the stability of the mobile Web, update frequency of the mobile Web, and performance criteria for large-scale mobile Web using top pages, returned HTTP codes and execution timestamps from the observation. The author discusses some of the mobile-specific issues in returned HTTP codes and time-outs. The author presents the current status of mobile Web in a long-term observation perspective