Transformer-based model for moroccan Arabizi-to-Arabic transliteration using a semi-automatic annotated dataset

Abstract

Language models have recently achieved state-of-the-art results in tasks such as translation, sentiment analysis, and text classification for high-resource languages. However, dedicated models for low-resource languages remain scarce, largely due to a lack of annotated data and linguistic resources. Most efforts focus on fine-tuning models trained on high-resource languages using limited data, resulting in a substantial performance gap. Moroccan Darija (MD), widely spoken in Morocco, lacks language resources and dedicated models. Additionally, MD texts often employ the Arabizi writing form, which combines Latin characters and numbers with Arabic script, further complicating Natural Language Processing (NLP) tasks. This work presents the first transformer-based model designed specifically for transliterating Moroccan Arabizi to Arabic. The approach leverages a character-level modeling architecture and a semi-automatically generated dataset containing over 33k word pairs, capturing significant linguistic diversity. The model achieves a state-of-the-art word transliteration accuracy (WTA) of 93 % and a character error rate (CER) of 4.73 % on unseen Moroccan Arabizi data, highlighting the potential of transformer models to improve transliteration accuracy for low-resource languages, particularly MD.