A Graph-Based Bursty Topic Detection Approach in User-Generated Texts

Abstract

The problem of hot bursty topic detection in user generated texts deserves great attentions with the proliferation of Internet technologies. However, traditional document clustering and probabilistic topic models that were developed for formal news articles are less effective for informal user-generated corpora. In this paper, we provide a graph-based perspective that well reflects the latent pattern of bursty topics in text stream and develop an effective solution of the bursty topic detection problem. We represent texts with topics using a directed and weighted graph, with the bursty words as vertices and Tversky index of bursty words being edges. Topic detection from the texts is then converted into dividing the constructed graph into separate sub graphs, each significant sub graph corresponding to a bursty topic. To accomplish this, we partition the bursty word graph into the graph's strongly connected components, based on the analysis that the important topical words within a graph are connected to each other with high weights and thus form strongly connected components. We demonstrate through experiments on two user-generated corpora collected from English web log and Chinese weibo (microblog) sites that the proposed approach can effectively detects the hot bursty topics, more appropriate than other topic detection models such as the LDA topic model and the EGF approach in TDT project.