Dynamic BI-Handed Sign Recognition Using 3D-CNN and Gated Recurrent Unit for Indian Sign Language

Abstract

Sign Language recognition systems play a pivotal role in fostering inclusivity and accessibility for the hearing and speech impaired. In this paper, we present a novel approach to Indian Sign Language recognition (ISL) recognition using deep learning techniques. Leveraging a self-created comprehensive dataset of dynamic ISL gestures, each represented as videos, created by volunteers under the supervision of experts, we propose a robust model architecture tailored to the unique characteristics of ISL. The emergence of convolutional neural networks (CNNs) and sequence-based models over the past decade has propelled the advancement of Sign Language Recognition (SLR) through artificial intelligence (AI). However, the domain of Indian Sign Language (ISL) has remained relatively under explored, primarily due to the absence of a comprehensive open-source dataset. Addressing this gap, our research endeavours to fill this void by conducting a thorough investigation and implementation of spatial and temporal feature-based methodologies on a newly created dataset. We utilized Mediapipe to extract spatial features. These features were then seamlessly integrated into a sequence-based gated recurrent unit (GRU) model, ensuring a robust framework for analysis. For the temporal dimension, we adopted a sophisticated sliding window approach, facilitating the training of three distinct convolutional neural network (CNN) models on a subset encompassing 21 dynamically bi-handed signed words. This work delineates the architectural intricacies underpinning each methodological approach, complemented by a comprehensive exposition of our newly recorded dataset. On the stated dataset and GRU technique using key-point method, we achieved the accuracy of 99.71%. Furthermore, we present an in-depth comparison of the accuracy attained by each technique employed, contributing to a nuanced understanding of their respective capabilities within the domain of Indian Sign Language recognition.