Towards scalable topic detection on web via simulating Lévy walks nature of topics in similarity space

Abstract

Organizing a few webpages from social media into hot topics is one of the key steps to understand trends on web. Discovering popular yet hot topics from web faces a sea of noise webpages which never evolve into popular topics. In this paper, we discover that the similarity values between webpages in a popular topic contain the statistically similar features observed in Lévy walks. Consequently, we present a simple, novel, yet very powerful Explore-Exploit (EE) approach to group topics by simulating Lévy walks nature in the similarity space. The proposed EE-based topic clustering is an effective and efficient method which is a solid move towards handling a sea of noise webpages. Experiments on two public data sets demonstrate that our approach is not only comparable to the State-Of-The-Art (SOTA) methods in terms of effectiveness but also significantly outperforms the SOTA methods in terms of efficiency.