Automatic detection of English words in Benglish text: A statistical approach

Abstract

Code-mixing and code-switching create challenges in the field of natural language processing applications like Machine Translation and Speech-to-Speech Translation. Detection of foreign words is very much essential for smooth processing of natural language. A statistical language independent approach for automatic detection of foreign words in mixed language has been introduced in this paper. Initially, the proposed approach has been applied on Benglish text which is combination of Bangla text contains English words. The methodology can be easily adopted for other languages where such code mixing exists. The proposed approach yields an accuracy of 71.82% when tested on sentences collected from Bangla blogs and social networking websites.