Identifying and Analyzing Reduplication Multiword Expressions in Hindi Text Using Machine Learning

Abstract

The task of identifying and analyzing Reduplication Multiword Expressions (RMWEs) in Natural Language Processing (NLP) involves extracting repeated words from various text forms and classifying them into Onomatopoeic, non-Onomatopoeic, partial, or semantic types. With the increasing use of low-resource languages in news, opinions, comments, hashtags, reviews, posts, and journals, this study proposes a machine learning-based RMWE identification method for Hindi text. The method employs linguistic patterns and statistical data, along with a proposed threshold boundary detection in statistical filtering. The Jaccard distance of dissimilarity and Sorensen Dice Coefficient of Similarity are used for semantic relation analysis. The proposed approach was evaluated using the publicly available Hindi corpus from IITB, measuring performance between two consecutive thresholds with the lowest error and highest recall. This study proposes an effective method for Indian computational linguistics, with experimental results highlighting its viability and utility, and providing a blueprint for current procedures.