Sentiment Analysis of Reviews in Kazakh With Transfer Learning Techniques

Abstract

Heavily pretrained transformer models, such as Bidirectional Encoder Representations from Transformers (BERT) or Generative Pre-trained Transformer (GPT), have successfully demonstrated the superior ability to recognize the right sentiments of texts in English or other dominant languages. However, for low-resource languages such as Kazakh, there are no similar models due to the high computational and memory requirements for their training and the lack of labeled datasets. Under this circumstance, transfer learning can be applied to low-resource language using a pretrained multilingual or related-language model. In this paper, we consider two ways to implement the transfer learning strategy: zero-shot learning and fine-tuning. We design experiments to compare these two methods and report the obtained results. Experiments show that in both cases BERT-based multilingual sentiment analysis model performs better than the BERT-based model for Turkish language, and the performance of these models grows after fine-tuning even with a very small number of samples in Kazakh.