Braille Recognition Based on a Dual-Mode Tactile Sensor With Piezoresistive and Piezoelectric Properties by the CNN-ResNet-BiLSTM Fusion Model

Abstract

Skin-like, flexible tactile sensors play a crucial role in healthcare and human-computer interaction. Based on multiwalled carbon nanotube (MWCNT)/cotton fabric (CF) piezoresistive sensor and polyvinylidene fluoride (PVDF) piezoelectric sensor, a dual-mode tactile sensor (MCP-DTS) featuring high sensitivity, excellent synergistic response, and stability is fabricated in conjunction with finite element analysis. The sensor is affixed to the stepper and slides uniformly across 25 different Braille character texture boards. Then, 3000 sets of sequential voltage data with 3500 dimensions and two channels collected by the sensor are used to form a dataset. On this basis, a convolutional neural network (CNN)-residual network (ResNet)-bidirectional long short-term memory (BiLSTM) fusion model combining CNN, ResNet, and BiLSTM is developed. This model demonstrates a robust feature extraction capability, achieving a high recognition accuracy (97.17%) for 25 different types of Braille. To verify the actual performance of the sensor, it is installed on the index finger to simulate the experience of a visually impaired person swiping to read Braille. Subsequently, the fusion model achieves high classification accuracy (89.17%) for Braille tactile perception. The MCP-DTS presented in this article demonstrates exceptional capability in perceiving tactile information and can effectively distinguish and recognize various types of tactile signals in Braille.