Enhancing topic coherence and diversity in document embeddings using LLMs: A focus on BERTopic

Abstract

With the rapid growth of digital textual data, the need for systematic organization of large datasets has become critical. Topic modeling stands out as an effective approach for analyzing large volumes of text datasets. Neural Topic Models (NTMs) have been developed to overcome the limitations of traditional methods by using contextual embeddings, such as Bidirectional Encoder Representations from Transformers (BERT), to improve topic coherence. Recent advancements in Natural Language Processing (NLP) have further enhanced document processing capabilities through large language models (LLMs) such as LLaMA and the Generative Pre-trained Transformer (GPT). This research explores whether LLM embeddings within NTMs offer better performance compared to conventional models like Sentence-BERT (S-BERT) and DistilBERT. In particular, we examine the impact of text preprocessing on topic modeling. A comparative analysis is conducted using datasets from three domains, evaluating six topic models, including LLMs such as Falcon and LLaMA3, using three evaluation metrics. Results show that while no single model consistently excelled across all metrics, LLaMA3 demonstrated the best performance in coherence among the LLMs. In addition, overall topic modeling performance improved with the application of all six preprocessing techniques. LLaMA3 showed progressively better performance with additional preprocessing, confirming its stability and effectiveness in topic modeling. These findings suggest that LLMs can be reliable tools for topic identification across diverse datasets.