Forum topic detection based on hierarchical clustering

Abstract

Forum has become one of the main platforms for people to express their personal point of view, with a lot of information surging in the forum everyday. How to detect automatically a forum topic among the massive information becomes an important and hard task. Though there are plenty of studies for topic detection, it is still a challenge to make it fast and accurately. This paper introduces the principle of maximum entropy and information gain when calculating feature weight. Our algorithm is based on the agglomerative hierarchical clustering (AHC). Experiments are focused on a game forum and handling sparse forum short texts. The result shows that the improved method can detect the forum topic more effectively.