A Modular Deep Learning Framework for Scene Understanding in Augmented Reality Applications

Abstract

Taking as input natural images and videos, augmented reality (AR) applications aim to enhance the real world with superimposed digital contents, enabling interaction between the user and the environment. One important step in this process is automatic scene analysis and understanding, which should be performed both in real time and with a good level of object recognition accuracy. In this work, an end-to-end framework based on the combination of a Super Resolution network with a detection and recognition deep network has been proposed to increase performance and lower processing time. This novel approach has been evaluated on two different datasets: the popular COCO dataset, whose real images are used for benchmarking many different computer vision tasks, and a generated dataset with synthetic images recreating a variety of environmental, lighting, and acquisition conditions. The evaluation analysis is focused on small objects, which are more challenging to correctly detect and recognise. The results show that the Average Precision is higher for small and low-resolution objects for the proposed end-to-end approach in most of the selected conditions.