Empirical Study on Efficiency of Different Language Modeling Techniques using Masking of Named Entities for Indic Languages

Abstract

Processing unstructured text in Natural Language Processing (NLP) poses significant challenges for Indic languages, which feature flexible word order, spelling variations, and complex sentence structures. Traditional models often struggle with these complexities, leading to issues such as out-of-vocabulary (OOV) words and increased perplexity. Neural Language Models (NLMs), particularly transformer-based models, address some of these challenges by employing word representations and self-attention mechanisms. However, OOV problems persist, especially with named entities, which are dynamic and vary across domains, making it difficult to create comprehensive lists of names for people, organizations, and locations. To address this, the Masked Entity-Based Language Model (ME-LM) has been introduced, focusing on masking named entities identified through Named Entity Recognition (NER) using pre-trained models like BERT-base-NER and IndicNER. Applied to Indic languages such as Hindi, Kannada, and Telugu for the first time, ME-LM has significantly reduced OOV occurrences by 18.60% to 94.70% and lowered perplexity. Since this is the first application of ME-LM to these languages, no standard benchmark exists for direct comparison, but the results show strong potential for improving named entity handling in these languages.