Parallelizing hot topic detection of microblog on spark

Abstract

With the emergence of the big data age, how to get valuable hot topic from the vast amount of digitized textual materials quickly and accurately has attracted more and more attention. This paper proposes a parallel Two-phase Mic-mac Hot Topic Detection(TMHTD) method specially design for microblogging in “Big Data” environment, which is implemented based on Apache Spark cloud computing environment. TMHTD is a distributed clustering framework for documents sets with two phases, including micro-clustering and macro-clustering. In the first phase, TMHTD partitions original data sets into a group of smaller data sets, and these data subsets are clustered into many small topics, producing intermediate results. In the second phase, the intermediate results are integrated into one, further clustered, and achieve the final hot topic sets. To improve the accuracy of the hot topic detection, an optimization of TMHTD is proposed. To handle large databases, we deliberately design a group of MapReduce jobs to concretely accomplish the hot topic detection in a highly scalable way. Extensive experimental results indicate that the accuracy and performance of TMHTD algorithm can be improved significantly over existing approaches.