Semantic Segmentation based Hand Gesture Recognition using Deep Neural Networks

The ability to discern the shape of hands can be a vital issue in improving the performance of hand gesture recognition. Segmentation itself is a very challenging problem having various constraints like illumination variation, complex background etc. The objective of the paper is to incorporate the perception of semantic segmentation into a classification problem and make use of the deep neural models to achieve improved results. This paper utilizes the UNET architecture to obtain the semantically segmented mask of the input, which is then given to a VGG16 model for classification. Here the top classifier layer of the VGG16 model is replaced with a classifier designed specifically for classifying the gestures at hand. The Brazilian Sign Language database used in the paper contains about 9600 images. Data augmentation process is used in preprocessing to generate sufficient number of training images for the aforementioned CNN-based models. A significant and improved average recognition rate of 98.97% is achieved through inherent feature learning capability of CNN and refined segmentation for 34 classes.