Research on Chinese Short Text Classification Based on Pre-trained Hybrid Neural Network

Abstract

Traditional text classification models mostly use the Word2vec and Glove to represent word vectors. When these traditional models classify Chinese short text data, they cannot well represent contextual semantic relationships and cannot completely extract text features. In this paper, the ERNIE (Enhanced Representation through Knowledge Integration) model is applied to the hybrid neural network model, which enhances the semantic representation of characters and generates character vectors by associating context semantic relations. Then the CNN (Convolutional Neural Network) and BiLSTM (Bidirectional Long Short Term Memory) are applied to the hybrid neural network to extract the characteristic information of the text data through CNN's different size convolution kernel and BiLSTM's bidirectional network structure. Moreover, in the training process, the weight decay mechanism of the AdamW algorithm is used to replace the traditional Adam algorithm to optimize the model performance. Finally, the obtained classification results are output by softmax classifier. By setting up comparative experiments on the THUCNews dataset and TouTiaoNews dataset, the results show that the Precision, Recall and F1-score of this model have been effectively improved over traditional neural network model and BERT-based model.