A remote conversation support system for deaf-mute persons based on bimanual gestures recognition using finger-worn devices

Communication between deaf-mute persons and normal people is very difficult when the subjects are out of the objects' view side, or vice verse. This paper presents a hand gesture recognition system using Magic Ring (a ring-shape wearable device) instead of data glove or camera to translate the predefined hand gestures into Japanese and to accomplish remote transmission of the information/command between user groups. The hand gestures are defined based on simplified Japanese sign language. The KNN method is adopted to the pattern recognition according to the bimanual sensing data. To improve the recognition rate, three most similar pre-reserved gestures and similarity are recorded, and Euclidean distance of feature values is used to estimate the gesture similarity. According to the user survey, the system shows good usability and most of the users would like to use such a system. Moreover, in the user-dependent setting, the bimanual accuracy is 85.2%, which is our performance 7.8% by using only right hand.