Arabic Sign Language Detection Using Deep Learning Based Pose Estimation

Abstract

It is necessary to determine if the person is signing or not and if the type of sign is static or dynamic when processing a series of video frames captured by the camera. The benefits of sign detection are: First, whether there is a sign to be recognized. Second, in the case of a static sign, only one frame should be used for sign recognition, while in the case of a dynamic sign, a series of frames should be used for sign recognition. The presented research aims to develop a model for a detect signer in a video stream for Arabic sign language and classify the signs among static, dynamic, and non-sign. A large dataset is needed to identify signs and get better results. Seven thousand five hundred videos were captured and collected for this purpose. The proposed system extracts keypoints of human poses in video frames using the MediaPipe library. Then it uses these keypoints to compute important features (distance and angles), training an Bidirectional Gated Recurrent Unit (BiGRU) model with those features to detect Sign Language of 99% test accuracy in real-time.