Enhancing Graph Variational Autoencoder for Short Text Topic Modeling with Mutual Information Maximization

Abstract

Neural topic models can successfully capture thematic patterns of the document with black-box variational inference, but they still suffer from sparsity problem when facing short texts with limited contextual information. To alleviate the sparsity problem, some graph-based methods have been proposed to explicitly model the word co-occurrence patterns. However, they ignore sequential information and word relevance degree in the document, resulting in inaccurate topic representations. Therefore, we propose a novel graph-based neural topic model, namely mutual Information enhanced Graph Topic Model (InfoGTM), which leverages the sequential information and takes into account the word relevance degree into topic modeling using a more accurate semantic graph. More specifically, instead of pre-computing statistical word co-occurrence, we develop an automatic way to dynamically construct semantic graph with a multi-head attention mechanism, which integrates both contextual and words structure information into the semantic graph, thereby providing more accurate word co-occurrence information. After that, a graph variational auto-encoder topic modeling framework is adopted to generate topic proportions for each short text. To further enhance the topic representation, we maximize the mutual information between input words and topic representations to ensure more semantic information could be compressed. Besides, mutual information maximization could preserve the smooth manifold structure of short texts, which enables the spread the similar topic representation from neighboring short texts. Substantial experiments are conducted on several benchmark data sets that verify the superiority of our method compared to the state-of-the-arts regard to the topic coherence performance.