Topic Detection in Chinese Microblogs Using Hot Term Discovery and Adaptive Spectral Clustering

Abstract

Weibo is a popular Chinese microblogging service that counts with millions of users and allows them to share short text messages. As an information network, Weibo can tell people what they care about as it is happening in the society. Unfortunately, users are constantly struggling to keep up with the larger and larger amounts of messages published every day. In order to help users to get the big picture, an efficient and effective topic detection method is urgent in demand. Considering the sheer scale and rapid evolution of the microblog messages, we investigate a novel method for topic detection in Chinese Microblogs in a given time period. It is composed of two major steps. First, hot terms are extracted by a suffix array structure and a TF*SDF term weighting scheme. Second, based on the extracted hot terms, we calculate their co-occurrence information and then group the terms into clusters that represent topics using an adaptive spectral clustering. Extensive experimental results on real world data demonstrate that the proposed method is more effective and efficient for topic detection in Chinese microblogs than existing approaches.