Part-Of-Speech Tagger in Malayalam Using Bi-directional LSTM

Abstract

The majority of activities performed by humans are done through language, whether communicated directly or reported using natural language. As technology is increasingly making the methods and platforms on which we communicate ever more accessible, there is a great need to understand the languages we use to communicate. By combining the power of artificial intelligence, computational linguistics and computer science, natural language processing (NLP) helps machines “read” text by simulating the human ability to understand language. Part-of-speech tagging (POS Tagging) is done as a pre-requisite to simplify a lot of different NLP applications like question answering, speech recognition, machine translation, and so on. Here, we attempt a comparison between part-of-speech taggers in Malayalam using decision tree algorithm and bi-directional long short term memory (BLSTM). The experiments presented in this paper use two corpora, one of 29076 sentences and the other of 500 sentences for performance evaluation. The experiments demonstrate the potential of architectural choice of BLSTM-based tagger over conventional decision tree-based tagging in Malayalam.