Automatic Hand Gesture Recognition with Semantic Segmentation and Deep Learning

Abstract

Automatic Hand Gesture Recognition is a key requirement for variety of applications, including translation of Sign Language, Human-Computer Interaction (HCI) and, ubiquitous vision-based systems. Due to the lighting variance and complicated background in the input image set of gestures, meeting this criterion remains a challenge. This paper introduces semantic segmentation to deep learning-based hand gesture recognition system for sign language translation. Building on the U - Net architecture, the proposed model obtains the semantically segmented mask of the input image, which is then fed to convolutional neural networks (CNNs) for multiclass classification. The proposed model is trained and tested for four different depths of the CNN architectures followed by the comparison with some pre-trained CNN architectures such as Inception V3, VGG16, VGG19, ResNet50. The proposed model is evaluated on National University of Singapore (NUS) hand posture dataset II (subset A), which contains 2000 images in 10 classes. A significant recognition rate of 97.15 % is achieved for the proposed model outperforming a set of prominent models and demonstrating its promises for sign language translation.