A Replacement Algorithm Designed for the Web Search Engine and Its Application in Storage Cache

With popularity of different kind of search engines on WWW, it requires the backend storage system to provide better physical I/O performance to speedup the query service perceived by end users. However, existing general purpose designed replacement algorithm can’t performs well for the web search applications. This paper first studies the access pattern of various real-life web search workload and then propose a new replacement algorithm RED-LRU based on the observed access properties. The simulation results shows that our proposed algorithm uniformly outperform the other replacement algorithms for all the workloads and cache size. To validate the simulation results, we integrate RED-LRU algorithm into a real storage cache DPCache. The experiment results in real system confirm the effectiveness of our proposed algorithm in improving the caching performance for web search application. Moreover, the runtime overhead of RED-LRU is also fairly low in practice.