Word-level Language Identification and Localization in Code-Mixed Urdu-English Text

Abstract

Language Identification is significant for most Natural Language Processing (NLP) tasks to work precisely. Language Identification is still very challenging because of the range of dialects. The major challenge in Language Identification (LID) task is the lack of availability of tools for understanding the context of multiple languages. We proposed a deep learning neural network Bi-LSTM CNN for word-level classification for Language Identification (LID) and localization of Roman Urdu and English in the code-switch text in this paper. We utilized the dataset of code-switch text having variant spellings of the same Roman Urdu words, generated from different social media platforms as they are a rich source of code-switch languages. We used GoogleNews Word2Vec Vectorizer for word embeddings. The embedding layer is followed by the Bidirectional long-short term memory (Bi-LSTM) layers along with the Convolutional Neural Network (CNN). We experimented with the dataset on different variations of LSTM and CNN to achieve the best possible results. We achieved 90.40% accuracy and a 90.39% F1 score.