A study of Chinese Text Classification based on a new type of BERT pre-training

Abstract

Chinese Text Classification (TC) is the process of mapping text to a pre-given topics category. In recent years, it has been found that TC is mainly based on RNN and BERT, so the development of different novel pre-training applied to Chinese TC is described as based on BERT pre-training. BERT combined with convolutional neural network is proposed to extend the BERT-CNN model for the problem of lack of semantic knowledge of BERT to derive a good classification effect. The second RoBERTa model performs feature extraction and fusion to obtain the feature word vector as the text output vector, which can solve the problem of insufficient BERT extracted features. The BERT-BiGRU model, on the other hand, mainly avoids the increase in the number of texts leading to long training time and overfitting, and uses a simpler GRU bi-word network as the main network to fully extract the contextual information of Chinese texts and finally complete the classification of Chinese texts; at the same time, it makes an outlook and conclusion on the new pre-training model for Chinese TC.