Development of word count data corpus for Hindi and Marathi literature

Abstract

India has a huge diversity of languages, and Hindi and Marathi are the most spoken languages in the northern and western parts of India. Hindi and Marathi have more than 528 million and 83 million speakers, respectively. The paper describes the development of the Hindi Word Corpus (Hindi WordCorp) and the Marathi Word Corpus (Marathi WordCorp), reporting the frequency of single words (1-gram) used in written texts of the respective languages using the bag-of-words model (BoW). The word frequencies are provided for eleven decades (pre-1920, 1920 to 2020). These texts include books (fiction, non-fiction, history, autobiographies, etc.) and magazines. Academic and reference books were not used. The Hindi WordCorp and Marathi WordCorp used 640 and 712 texts, respectively. An analysis was employed to check whether the texts used were enough to stabilize the rank-order of the total frequencies of the words. Zipf's and Heaps’ law coefficients indicated the sufficiency of the texts. Researchers in various areas like linguistics, social sciences, text mining, machine learning, etc., can use the dataset to answer research questions about language and culture. Some demonstrative examples are provided for using the datasets in the two languages. The dataset is made available on an open data repository. The paper is an account of dataset creation for Hindi and Marathi WordCorp. Hence, no empirical results or conclusions are made based on the data created. A WebApp named Indian Languages Word Corpus (ILWC) has been developed for users. Future directions for text mining and language models are discussed.