A Robot-based Arabic Sign Language Translating System

Abstract

Services provided to deaf people in the Eastern province of Saudi Arabia were evaluated, which confirmed a high need to support the deaf community. This paper proposes utilizing the Pepper robot in the task of recognizing and translating Arabic sign language (ArSL), by which the robot recognizes static hand gestures of the letters in ArSL from each keyframe extracted from the input video and translate it into written text and vice versa. This project aims to conduct a two-way translation of the Arabic sign language in a way that fulfills the communication gap found in Saudi Arabia among deaf and non-deaf people. The methods proposed in this paper are computer vision to use the pepper robot's camera and sensors, Natural language processing to convert natural speech to sign language and Deep learning to build a convolutional neural network model that classifies the sign language gestures and convert them into their corresponding written and spoken form. Moreover, two datasets were used, first one is a collection of hand gestures for training the model and the other one is 39 animated signs of all the Arabic letters and special letters.