Hand Gesture Recognition of Hand Shapes in Varied Orientations using Deep Learning

Abstract

A large number of Deaf people are unable to communicate by means of spoken language. Thus, a translation system that converts South African Sign Language to English and vice versa would be invaluable to the Deaf community. In order to recognise sign language gestures, five fundamental gesture parameters, namely, hand shape, hand orientation, hand motion, hand location and facial expressions, need to be recognised separately. The research in this paper aims to utilise deep learning techniques, specifically convolutional neural networks, to recognise a set of hand shapes in various orientations within a live video stream captured on an iPhone mobile device. The research forms part of a larger project that aims to automatically translate South African Sign Language into English and vice versa. The research proposed two approaches for classifying gestures, a two stage approach and single classifier approach. The former approach managed to achieve an average accuracy of 70% and the latter an average accuracy of 67%.