Visual and tactile perception techniques for braille recognition

Abstract

In the case of a visually impaired person, literal communication often relies on braille, a system predominantly dependent on vision and touch. This study entailed the development of a visual and tactile perception technique for braille character recognition. In the visual perception approach, a braille character recognition was performed using a deep learning model (Faster R-CNN–FPN–ResNet-50), based on custom-made braille dataset collected through data augmentation and preprocessing. The attained performance was indicated by an mAP50 of 94.8 and mAP75 of 70.4 on the generated dataset. In the tactile perception approach, a braille character recognition was performed using a flexible capacitive pressure sensor array. The sensor size and density were designed according to braille standards, and a single sensor with a size of 1.5 mm × 1.5 mm was manufactured into a 5 × 5 sensor array by using a printing technique. Additionally, the sensitivity was improved by incorporating a pressure-sensitive micro dome-structured array layer. Finally, braille character recognition was visualized in the form of a video-based heatmap. These results will potentially be a cornerstone in developing assistive technology for the visually impaired through the fusion of visual-tactile sensing technology.