Multi-Scale Spatial-Angular Collaborative Guidance Network for Heterogeneous Light Field Spatial Super-Resolution

Abstract

Light Field (LF) imaging captures the spatial and angular information of light rays in the real world and enables various applications, including digital refocusing and single-shot depth estimation. Unfortunately, due to the limited sensor size of LF cameras, the captured LF images suffer from low spatial resolution while providing a dense angular sampling. Existing single-input LF spatial super-resolution (SR) methods usually utilize the inherent sub-pixel information to recover high-frequency textures, but they struggle in large-scale SR tasks (e.g., $8\times $ ). Conversely, the heterogeneous imaging approach combining an LF camera and a 2D digital camera can capture richer information for effective large-scale reconstruction. To this end, this paper proposes a multi-scale spatial-angular collaborative guidance network (LF-MSACGNet) for heterogeneous LF spatial SR. Specifically, a context-guided deformable alignment module is first designed, which utilizes high-level feature information to achieve precise alignment between the low-resolution LF image and the 2D high-resolution image. Subsequently, a Transformer-driven spatial-angular collaborative guidance module is constructed to explore the spatial-angular correlation and complementarity. This allows for an effective fusion of the multi-resolution spatial-angular features. Finally, the SR LF image is reconstructed through a spatial-angular aggregation module. In addition, a multi-scale training strategy is adopted to subdivide the challenging large-scale SR task into multiple simple tasks to boost the SR performance. Experimental results on seven public datasets show that the proposed method outperforms the state-of-the-art SR methods in both quantitative and qualitative comparison, and exhibits favorable robustness to wide baseline LF images.