Tweet Sentiment Extraction Using Byte Level Pretrained Language Model∗

Research on sentiment analysis developed rapidly in recent years, and twitter sentiment analysis is one of the most popular topics. Besides classifying the sentiment, it is also important to find out the decisive phrases or words of the text to the classified sentimental category. In this paper, we proposed and developed byte-level pretained RoBERTa models, they are designed to extract phrases from tweet data with sentiment labels. We compared RoBERTa model and its’ variants, including RoBERTa-base, RoBERTa-large, XLM-RoBERTa-base, and RoBERTa-large-mnli. We build the model with RoBERTa model and CNN, then train the model with given tweet text and sentiment labels so that the deciding phrases of sentiments can be predicted. Our results show that RoBERTa-base obtains Jaccard score of 0.712 and training time of 240 minutes in total, which is the best performance among all the models.