American Sign Language Fingerspelling Recognition using Attention Model

Abstract

Sign Language Recognition(SLR) is a complex gesture recognition problem because of the quick and highly coarticulated motion involved in gestures. This research work focuses on Fingerspelling recognition task, which constitutes 35% of the American Sign Language (ASL). Fingerspelling identifies the word letter by letter. Fingerspelling is used for signing the words which do not have designated ASL signs such as technical terms, content words and proper nouns. In our proposed work for ASL Fingerspelling recognition, we consider ChicagoFSWild dataset which consists of occlusions and images captured in varying illuminations, lighting conditions (in the wild environments). The optical flow is obtained from Lucas-Kanade algorithm, prior is generated, images are resized and cropped with face-roi technique to get the region of interest (ROI). The visual attention mechanism attends to the ROI iteratively. ResNet, pretrained on Imagenet is used for the extraction of spatial features. The Bi-LSTM network with Connectionist Temporal Classification (CTC) is used to predict the sign. It provides the accuracy of 57% on ChicagoFSWild dataset for Fingerspelling recognition task.