Rating the Severity of Toxic Comments Using BERT-Based Deep Learning Method

With the integration of Internet and smartphones into people’s lives, toxic comments become ubiquitous on various online social media. These comments hinder seriously the construction of a safe and healthy network environment, leading to a great demand for automated methods which can effectively identify such harmful information and deal with it in a timely manner. To address this challenge, we propose a BERT-based deep learning method in this paper to rate the severity of toxic comments. On the basis of the text dataset provided by Jigsaw, BERT-based backbones (RoBERTa and DeBERTa) are trained to extract contextualized embeddings from sentences. After that, corresponding severity scores of comments are calculated by the subsequent head layers, where the head is chosen from the multilayer perceptron, convolutional neural network, and attention structure. After applying the K-Fold cross validation and an average ensemble of different models, our method achieves a rank 28/2301 (top 1.2%) in the leaderboard of Jigsaw Rate Severity of Toxic Comments Kaggle competition. This result can get a silver medal in this competition, and proves that our model can be an effective approach to rate precisely the severity of a toxic comment. This work can remarkably reduce the workload of manual review of Internet content and help build a more harmonious online community environment.