Enhancing neural topic modeling for social media text via semantic bag of word clusters and log-domain Sinkhorn transport

Abstract

Topic modeling has been widely applied to analyze text data from social media platforms. Under this scenario, traditional Neural Topic Models (NTMs) encounter three primary challenges: (1) initial text representation; (2) the long-tail nature of topic distributions in social network texts; (3) approximation of Optimal Transport. Motivated by these challenges, we propose an end-to-end solution spanning from text representation to topic modeling.

First, we propose SBoWC, a novel text representation method that performs dimensionality reduction while absorbing semantic information through base terms, achieved by combining word embeddings with clustering statistics. Subsequently, we propose GSWTM, a Wasserstein-based autoencoder topic model that fits the long-tail topic distribution in social network texts via Gamma priors and innovatively employs log-domain Sinkhorn to approximate Optimal Transport.

Ablation studies demonstrate the transferability and effectiveness of SBoWC in text representation. GSWTM demonstrates significantly better performance than baselines in TU, $C_V$, and the comprehensive metrics TQ across four real social network datasets of varying sizes. The log-domain Sinkhorn approximation exhibits excellent stability, allowing the regularization parameter $ϵ$ to be reduced to 0.1–0.01, thereby approaching the original Optimal Transport.