VSS-SAM++: Visual State Space-Aware SAM for 3-D Medical Image Segmentation

Abstract

The Segment Anything Model (SAM) has demonstrated groundbreaking performance in natural image segmentation, yet its direct application to medical imaging remains suboptimal due to domain shifts in data distributions and the inherent 3D nature of medical data. Although recent SAM-based methods have employed parameter-efficient transfer learning (PETL) to adapt SAM for medical tasks, they often overlook the critical 3D contextual information essential for accurate volumetric segmentation. To address this limitation, we introduce VSS-SAM++, a novel dual-branch architecture that combines SAM’s foundational visual priors with Vision Mamba’s capacity for modeling long-range spatial dependencies. In this framework, SAM serves as the primary encoder for high-level feature extraction, while a parallel Mamba branch captures cross-slice dependencies in 3D medical volumes. A gated hybrid attention module then dynamically fuses complementary features from both branches, adaptively weighting multi-view representations to minimize feature ambiguity and enhance segmentation precision. Extensive evaluations across nine public CT and MRI datasets demonstrate that VSS-SAM++ outperforms existing methods by 0.2–11.3% in Dice score on diverse organ segmentation tasks. The framework’s robustness to domain shifts and scalability across diverse modalities highlights its potential for clinical deployment.