OAFuser: Toward Omni-Aperture Fusion for Light Field Semantic Segmentation

Abstract

Light field cameras are capable of capturing intricate angular and spatial details. This allows for acquiring complex light patterns and details from multiple angles, significantly enhancing the precision of image semantic segmentation. However, two significant issues arise: 1) The extensive angular information of light field cameras contains a large amount of redundant data, which is overwhelming for the limited hardware resources of intelligent agents. 2) A relative displacement difference exists in the data collected by different microlenses. To address these issues, we propose an omni-aperture fusion model (OAFuser) that leverages dense context from the central view and extracts the angular information from subaperture images to generate semantically consistent results. To simultaneously streamline the redundant information from the light field cameras and avoid feature loss during network propagation, we present a simple yet very effective subaperture fusion module (SAFM) . This module efficiently embeds subaperture images in angular features, allowing the network to process each subaperture image with a minimal computational demand of only ( ${\sim}1\rm GFlops$ ). Furthermore, to address the mismatched spatial information across viewpoints, we present a center angular rectification module (CARM) to realize feature resorting and prevent feature occlusion caused by misalignment. The proposed OAFuser achieves state-of-the-art performance on four UrbanLF datasets in terms of all evaluation metrics and sets a new record of $84.93\%$ in mIoU on the UrbanLF-Real Extended dataset, with a gain of ${+}3.69\%$ . The source code for OAFuser is available at https://github.com/FeiBryantkit/OAFuser .