EI-RNN-based text generation for the static and dynamic isolated sign language videos

Abstract

SLR, which assists hearing-impaired people to communicate with other persons by sign language, is considered as a promising method. However, as the features of some of the static SL could be the same as the feature in a single frame of dynamic Isolated Sign Language (ISL), the generation of accurate text corresponding to the SL is necessary during the SLR. Therefore, Edge-directed Interpolation-based Recurrent Neural Network (EI-RNN)-centered text generation with varied features of the static and dynamic Isolated SL is proposed in this article. Primarily, ISL videos are converted to frames and pre-processed with key frame extraction and illumination control. After that, the foreground is separated with the Symmetric Normalised Laplacian-centered Otsu Thresholding (SLOT) technique for finding accurate key points in the human pose. The human pose’s key points are extracted with the Media Pipeline Holistic (MPH) pipeline approach and to improve the features of the face and hand sign, the resultant frame is fused with the depth image. After that, to differentiate the static and dynamic actions, the action change in the fused frames is determined with a correlation matrix. After that, to engender the output text for the respective SL, features are extracted individually as of the static and dynamic frames. It is obtained from the analysis that when analogized to the prevailing models, the proposed EI-RNN’s translation accuracy is elevated by 2.05% in INCLUDE 50 Indian SL based Dataset and Top 1 Accuracy 2.44% and Top 10 accuracy, 1.71% improved in WLASL 100 American SL.