Sign language localization: Learning to eliminate language dialects

Abstract

Machine translation of sign language into spoken languages is an important yet non-trivial task. The sheer variety of dialects that exist in any sign language makes it only harder to come up with a generalized sign language classification system. Though a lot of work has been done in this area previously but most of the approaches rely on intrusive hardware in the form of wired or colored gloves or are specific language/dialect dependent for accurate sign language interpretation. We propose a cost-effective, non-intrusive webcam based solution in which a person from any part of the world can train our system to make it learn the sign language in their own specific dialect, so that our software can then correctly translate the hand signs into a commonly spoken language, such as English. Image based hand gesture recognition carries sheer importance in this task. The heart of hand gesture recognition systems is the detection and extraction of the sign (hand gesture) from the input image stream. Our work uses functions like skin color based thresholding, contour detection and convexity defect for detection of hands and identification of important points on the hand respectively. The distance of these important contour points from the centroid of the hand becomes our feature vector against which we train our neural network. The system works in two phases. In the training phase the correspondence between users hand gestures against each sign language symbol is learnt using a feed forward neural network with back propagation learning algorithm. Once the training is complete, user is free to use our system for translation or communication with other people. Experimental results based on training and testing the system with numerous users show that the proposed method can work well for dialect-free sign language translation (numerals and alphabets) and gives us average recognition accuracies of around 65% and 55% with the maximum recognition accuracies rising upto 77% and 62% respectively.