SudaBERT: A Pre-trained Encoder Representation For Sudanese Arabic Dialect

Abstract

Bidirectional Encoder Representations from Transformers (BERT) has proven to be very efficient at Natural Language Understanding (NLU), as it allows to achieve state-of-the-art results in most NLU tasks. In this work we aim to utilize the power of BERT in Sudanese Arabic dialect, and produce a Sudanese word representation. We collected over 7 million sentences in Sudanese dialect and used them to resume training of the pre-trained Arabic-BERT, as it was trained on large Modern Standard Arabic (MSA) corpus. Our model -SudaBERT- has achieved better performance on Sudanese Sentiment Analysis, this clarifies that SudaBERT works better in understanding Sudanese Dialectic which is the domain we are interested in.