A Vision-based Hand-sign Language Teaching System using Deep Neural Network: Methodology and Experiments

Abstract

In this paper, a real-time hand-sign language teaching system using deep neural network is proposed. Communication presents a significant barrier for persons who are impaired in hearing and speaking. There are various projects and studies have been conducted to create or improve smart systems for this rapid-growth population. Deep learning approaches became widely used to enhance the accuracy of sign language recognition models. However, most research has primarily concentrated on hand gestures for translation, not language self-learning for long-term development. This work aims to construct a complete system to assist deaf and mute people in studying and examining their performance. First, we designed and built a prosthetic arm equipped with a monocular camera using 3D printing. Second, the MediaPipe library was used to extract key points from collected videos of the hand gestures. Then, the Gated Recurrent Units model is trained to recognize words based on the data. The real-time experimental results demonstrate the system's effectiveness and potential with 97 percent accuracy.