BiSign-Net: Fine-grained Static Sign Language Recognition based on Bilinear CNN

Abstract

Sign language (SL) is a type of communication language used by deaf and hard-of-hearing people. Large varieties in different SLs and lack of knowledge in general public to interpret them bring an inevitable necessity for breaking down the communication barriers by automatic sign language recognition (SLR) systems. Despite the existence of numerous approaches with satisfactory performance, they still suffer from severe challenges in dealing with large intra-class and slight inter-class variations, which make them infeasible for real-world applications. To address this issue, a novel end-to-end fine-grained static SLR (SSLR) system is proposed, namely BiSign-Net, based on Bilinear Convolutional Neural Network (Bi-CNN) to efficiently model the variations both in the location and appearance of the hands in the images for enhancing the accuracy, speed, and robustness against the translation. To this end, fine-grained orderless bilinear features are generated by pooled outer product of the extracted features from two identical novel CNN-based feature extractors. Bilinear features pass a normalization module including the signed square root and l2 normalization through which the accuracy of the model is further improved. A dropout layer is deployed in the classification module to aid the model in dealing with small-scale datasets by preventing overfitting. The number of layers, hyper-parameters, and optimization technique of the proposed CNN are adjusted to achieve high performance and faster convergence with low number of parameters. Experimental results on four datasets of Static ASL, NUS I, Massey, and ArASL from two SLs (i.e. American and Arabic) with an accuracy of 100%, 100%, 99.20%, and 99.35%, respectively, demonstrate that the proposed model surpasses the existing approaches with high robustness and generalization ability.