A framework for human-computer interaction using dynamic time warping and neural network

Due to incapability in speaking and hearing, there always exist a group of people in the community who face difficulties in communication. These people use some symbols and gestures to convey their messages and receive their messages, this form of communication is known as Sign Language. We have provided a solution based on dynamic time warping (DTW) for the first module and software based solution for the second module by exploiting the latest technology of Microsoft Kinect depth camera which tracks the 20 joint location of human beings. In sign to speech/text conversion block, the actor performs some valid gestures within the Kinect field of view. The gestures are taken up by the Kinect sensor and then interpreted by comparing it with already stored trained gestures in the dictionary. After the sign is recognized, it is copied to the respective word which is transferred to the speech conversion and text conversion module to produce the output. In the second block, which is a speech to sign/gesture conversion, the person speaks in Kinect field of view which is taken by the Kinect, and the system converts speech into text, and corresponding word is mapped into predefined gesture which is played on the screen. This way a disabled person can visualize the spoken word. The accuracy of sign to speech module is found to be 87%, and that of speech to gesture module is 91.203%.