An Interactive Device for Ambient Awareness on Sidewalk for Visually Impaired

Ambient awareness on a sidewalk is critical for safe navigation, especially for the people who are visually impaired. Awareness of obstacles such as debris, potholes, construction site, and traffic movement pattern improve mobility and independence of them. To address this problem, we implemented an interactive and portable sidewalk assistive device (IPSAD). The key idea is to use the power of deep learning to model the “obstacles” on a sidewalk to provide personalized feedback to the user. We focus on transfer learning and fine tuning of pre-trained Convolutional Neural Network (CNN) models for real-time obstacle recognition that can be deployed in small form factor devices. This approach also account for issues related to the execution time and energy efficiency. We empirically evaluate a number of state-of-the-art architectures to choose the best model based on fewer parameters and lower energy consumption. Finally, we built fully integrated IPSAD prototype on Raspberry Pi3 (RPi3). Audio feedback scheme were implemented to accommodate user preferences and personalization. We perform quantitative evaluation of the prototype system based on the accuracy of the model on Ambient Awareness on Sidewalk (AS) dataset, on-field system performance, data communication, response time (capturing images, recognition of obstacle and feedback), and failure points. Empirical evaluation showed that the accuracy of the model is 87% on the AS dataset and 78.75% on-field system. The system can operate either standalone mode when there is no Internet or cloud mode. In addition, the system can use human intelligence through crowd source and caregiver(s) to assist the users when automated system fails.