Gestural User Interfaces for Hearing and Speech Impaired People Using KINECT

Abstract

A large Communication gap exist between Hearing and Speech Impaired People(HSIP) and normal hearing people because HSIP used sign language and normal people used spoken language. Sign language is the only way to communicate with HSIP. There are many system exists that translate signs into text of spoken language to make possible of this communication. Our proposed system Home Sign Language Tutor is divided into two main sections. One is home sign dictionary that helps to learn the home sign language and other section translates the signs performed by HSIP into plain text in real time. We used kinect sensor to get skeletal data from user. User Centered Design process is used to design the system which involves user in every stage of the design process. After developing the system we performed an experiment on 30 (Thirty) HSIP. We divided the participants in 2 equal groups in which one was Normal and other is Expert. Normal group has hearing parents and expert group has hard hearing parents. Home signs are used in home in early age when HSIPs are not exposed to formal learning in school. In this proposed system we translated 21 (Twenty one) home signs into text. Almost 10 signs were already in use or have little bit understanding in HSIPs while we have designed 11 new signs with help of sign language interpreter. Results shows that 75 % participants from normal group and 60 % from expert group performed 11 to 15 gestures successfully. The same experiment is conducted again with same groups and success rate for 95% is achieved with normal group participants for 16 to 20 gestures. On the other hand 60 % expert group performed all 21 signs successfully.