Short Text Classification Model Based on BERT and Fusion Network

Abstract

Abstract: Aiming at short texts lacking contextual information, large amount of text data, sparse features, and traditional text feature representations that cannot dynamically obtain the key classification information of a word polysemous and contextual semantics. this paper proposes a pre-trained language model based on BERT. The network model B-BAtt-MPC (BERT-BiLSTM-Attention-Max-Pooling-Concat) that integrates BiLSTM, Attention mechanism and Max-Pooling mechanism. Firstly, obtain multi-dimensional and rich feature information such as text context semantics, grammar, and context through the BERT model; Secondly, use the BERT output vector to obtain the most important feature information worth noting through the BiLSTM, Attention layer and Max-Pooling layer; In order to optimize the classification model, the BERT and BiLSTM output vectors are fused and input into Max-Pooling; Finally, the classification results are obtained by fusing two feature vectors with Max-Pooling. The experimental results of two data sets show that the model proposed in this paper can obtain the importance and key rich semantic features of short text classification, and can improve the text classification effect.