Seeing eye drone: a deep learning, vision-based UAV for assisting the visually impaired with mobility

Seeing Eye Drone assists low-vision persons with environment awareness performing exploration and obstacle detection. The modalities of 3D (stereo) and 2D vision on a drone are compared for this task. Different deep-learning systems are developed including 2D only and 3D+2D networks. Comparisons of retrained networks versus training from scratch are also made and approximately 34,000 samples were collected for training and the resulting SSD CNN architecture is used to determine a user's location and direction of travel. A second network identifies locations of common objects in the scene. The object locations are then compared with the user location/heading and depth data to determine whether they represent obstacles. Obstacles determined to be in the user's region of interest are communicated to the visually-impaired user via Text-to-Speech. Real data from outdoor drone flights that communicate with an Android based application are shown.