Pseudo 5D hyperspectral light field for image semantic segmentation

Abstract

Light field (LF) encodes both intensity information and directional information of all light rays into high-dimensional signal, which facilitates various advanced applications due to its rich description. However, current mainstream research adopts two-plane parametrization to describe 4D LF, losing the information stored in the spectral dimension that can delineate more scene details. On this account, we introduce 5D hyperspectral light field (H-LF) to achieve robust semantic segmentation for the first time. To alleviate data redundancy while preserving useful information to a large extent, we use pseudo H-LF with sparsely non-repetitive angular-spectral distribution as an alternative and propose a network called PHLFNet. Specifically, our network successively performs feature-level angular-spectral joint blending and semantic-level angular-spectral joint enhancement to fully exploit the complementary information embedded in pseudo H-LF, in which the former executes preliminary information fusion and calibration across all modalities, and the latter distills unique semantic cues of each auxiliary modality to boost feature of segmented central view image. To guarantee the accuracy of semantic cues distillation, we design boundary consistency semantic label propagation to handle cross-spectral color inconsistency and cross-angular pixel misalignment in pseudo H-LF, thereby generating semantic labels of each auxiliary modality to provide supervision. Extensive experimental results illustrate that PHLFNet achieves outstanding performance compared with relevant state-of-the-art methods, demonstrating the significance of introducing H-LF for semantic segmentation.