Mamba4PASS: Vision Mamba for PAnoramic Semantic Segmentation

Abstract

PAnoramic Semantic Segmentation (PASS) is a significant and challenging task in the field of computer vision, aimed at achieving comprehensive scene understanding through an ultra-wide-angle view. However, the equirectangular projection (ERP) with richer contextual information is susceptible to geometric distortion and spatial discontinuity, which undoubtedly impede the efficacy of PASS. Recently, significant progress has been made in PASS, nevertheless, these methods often face a dilemma between global perception and efficient computation, as well as the effective trade-off between image geometric distortion and spatial discontinuity. To address this, we propose a novel framework for PASS, Mamba4PASS, which is more efficient compared to Transformer-based backbone models. We introduce an Incremental Feature Fusion (IFF) module that gradually integrates semantic features from deeper layers with spatial detail features from shallower layers, effectively alleviating the loss of local details caused by State Space Model (SSM). Additionally, we introduce a Spherical Geometry-Aware Deformable Patch Embedding (SGADPE) module, which leverages spherical geometry properties and employs a novel deformable convolution strategy to adapt to ERPs, effectively addressing spatial discontinuities and stabilizing geometric distortions. To the best of our knowledge, this is the first semantic segmentation model for panoramic images based on the Mamba architecture. We explore the potential of this approach for PASS, providing a new solution to this domain, and validate its effectiveness and advantages. Extensive experiments demonstrate the effectiveness of the proposed method, achieving state-of-the-art results compared to existing approaches.