Panoramic Depth and Semantic Estimation With Frequency and Distortion Aware Convolutions

Omnidirectional images reveal advantages when addressing the understanding of the environment due to the 360-degree contextual information. However, the inherent characteristics of the omnidirectional images add additional problems to obtain an accurate detection and segmentation of objects or a good depth estimation. To overcome these problems, we exploit convolutions in the frequency domain, obtaining a wider receptive field in each convolutional layer, and convolutions in the equirectangular projection, to cope with the image distortion. Both convolutions allow to leverage the whole context information from omnidirectional images. Our experiments show that our proposal has better performance on non-gravity-oriented panoramas than state-of-the-art methods and similar performance on oriented panoramas as specific state-of-the-art methods for semantic segmentation and for monocular depth estimation, outperforming the sole other method which provides both tasks.