SIT-SAM: A semantic-integration transformer that adapts the Segment Anything Model to zero-shot medical image semantic segmentation

Segment Anything Model (SAM) demonstrates zero-shot instance segmentation capabilities through prompt-guided interaction. However, its application to 3D medical imaging remains limited due to insufficient semantic understanding of complex anatomical structures. Current SAM variants attempt to address this challenge through architectural modifications and fine-tuning ; however, these approaches often compromise SAM’s original zero-shot capabilities. To bridge this gap, we introduce the semantic integration Transformer for SAM (SIT-SAM), an innovative post-processing framework that enhances SAM’s instance-level masks with semantic comprehension of anatomical structures. Our approach preserves SAM’s valuable zero-shot capabilities while introducing semantic awareness. Specifically, SIT-SAM comprises of three functional blocks: (1) the original SAM for instance mask generation, (2) a semantic integration transformer that combines hierarchical multi-scale feature extraction to capture both fine anatomical details and global context while leveraging instance mask geometry for enhanced anatomical structure understanding, (3) a cognitive science-inspired memory module for learning from limited training data. Evaluation on the TotalSegmentator dataset demonstrates SIT-SAM’s superior performance, achieving 90.55% accuracy, substantially outperforming the fine-tuned baseline i.e. SAM-Med3D with fully convolutional network (FCN) prediction head by 52.69%. SIT-SAM also exhibits robustness in data-constrained environments, delivering a 2.43% improvement with single-point prompt and maintaining effectiveness with multiple prompts, showing a 0.78% gain using ten point prompts. Code is available at https://github.com/wentao0429/SIT-SAM.