A Web Page De-duplication Algorithm Based on Data Clearing

Abstract

Duplicated web pages responded by search engines not only waste valuable storage, but also aggravate burdens of users’ browse. Web page de-duplication can effectively improve the information retrieval. This paper proposes pretreatment of web pages to improve the effectiveness and efficiency of web page de-duplication based on feature code according to the principle of data clearing. This paper features that ranking feature code to reduce the comparison times of the system and space and time complexity. Experiments show that this method has a promising prospect in eliminating large-scale duplicated web pages.