Multimodal hand/finger movement sensing and fuzzy encoding for data-efficient universal sign language recognition

Abstract

Wearable sign language recognition helps hearing/speech impaired people communicate with non-signers. However current technologies still unsatisfy practical uses due to the limitations of sensing and decoding capabilities. Here, A continuous sign language recognition system is proposed with multimodal hand/finger movement sensing and fuzzy encoding, trained with small word-level samples from one user, but applicable to sentence-level language recognition for new untrained users, achieving data-efficient universal recognition. A stretchable fabric strain sensor is developed by printing conductive poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS) ink on a pre-stretched fabric wrapping rubber band, allowing the strain sensor with superior performances of wide sensing range, high sensitivity, good linearity, fast dynamic response, low hysteresis, and good long-term reliability. A flexible e-skin with a homemade micro-flow sensor array is further developed to accurately capture three-dimensional hand movements. Benefitting from fabric strain sensors for finger movement sensing, micro-flow sensor array for 3D hand movement sensing, and human-inspired fuzzy encoding for semantic comprehension, sign language is captured accurately without the interferences from individual action differences. Experiment results show that the semantic comprehension accuracy reaches 99.7% and 95%, respectively, in recognizing 100 isolated words and 50 sentences for a trained user, and achieves 80% in recognizing 50 sentences for new untrained users.