Indian Sign Language Interpretation and Sentence Formation

Abstract

People with speech and hearing disabilities approximately constitute 1 percentage of the total Indian population. A person who is hearing and speech impaired is not able to compete or work with a normal person in a normal environment because of the lack of a proper communication medium.Sign Language is used for communication amongst them. Sign Language is the most natural and expressive way for the hearing and speech impaired. This paper proposes a method that recognizes Sign Language and converts it to normal text and speech for fast and improved communication amongst them and also with others. The focus is on the Indian Sign Language (ISL) specifically as there is no substantial work on ISL rendering the above requirements for these people.The paper focuses on developing a real-time hands-on system that takes video inputs of gestures in the specified ROI and performs gesture recognition using various feature extraction techniques and Hybrid-CNN model trained using the ISL database created. The correctly identified gesture tokens are sent to a Rule-Based Grammar and for Web Search query to generate various sentences and a Multi-Headed BERT grammar corrector provides grammatically precise and correct sentences as the final output.