Egocentric Scene Description for the Blind and Visually Impaired

Abstract

Image captioning methods come short when being used to describe scenes for the blind and visually impaired, because not only do they focus exclusively on salient objects, eliminating background and surrounding information, but they also do not offer egocentric positional descriptions of objects regarding the users, failing by that to give them the opportunity to understand and rebuild the scenes they are in. Furthermore, the majority of solutions neglect depth information, and models are trained solely on 2D (RGB) images, leading to less accurate prepositions and words or phrases' order. In this paper, we will offer the blind and visually impaired more descriptive captions for almost every region present in the image by the use of DenseCap model. Our contribution lies within the use of depth information that will be estimated by AdaBins model in order to enrich captions with positional information regarding the users, helping them understand their surroundings.