Pre-trained Contextualized Representation for Chinese Conversation Topic Classification

Abstract

Topic classification plays an important role in facilitating security-related applications, which can help people reduce data scope and acquire key information quickly. Conversation is one of the important ways of communication between people. The utterances in a conversation may contain vital clues, such as people’s opinions, emotions and political slants. To explore more effective approaches for Chinese conversational topic classification, in this paper, we propose a neural network architecture with pre-trained contextualized representation. We firstly apply pretrained BERT model to fine-tune and generate the conversational embeddings, which are the inputs of our neural network models. Then we design several models based on neural networks to extract task-oriented advanced features for topic classification. Experimental results indicate that the models based on our neural network architecture all outperform the baseline only fine-tuned with the pre-trained BERT model. It demonstrates that the pretrained representations are effective to Chinese conversational topic classification, and the proposed architecture can further capture the salient features from the representations. And we release the code and dataset of this paper that can be obtained from https://github.com/njoe9/pretrained_representation.