ProtoSAM-2D: 2D Semantic Segment Anything Model with Mask-Level Prototype-Learning and Distillation.

Abstract

Semantic segmentation on medical images has been marginally improved by deep learning methods with higher accuracy and efficiency in delineating anatomical structures and pathologies. However, traditional deep learning methods approaches have relied on fully supervised training using specific datasets on specific image modalities, limiting their adaptability across diverse medical imaging scenarios. The emergence of foundation models like the Segment Anything Model (SAM) has opened new avenues for interactive instance segmentation, but they lack semantic understanding, particularly in medical contexts where anatomical knowledge is important. To address this gap, we introduce ProtoSAM-2D, an enhancement of SAM-Med2D that integrates semantic capabilities into the interactive segmentation framework for 2D medical images. Our approach leverages a novel mask-level prototype prediction mechanism to generate and classify feature representations for each segmented instance by comparing them to learned prototypes. It enables efficient categorization of diverse anatomical structures and facilitates rapid adaptation to new classes. To optimize computational efficiency, we implement a distillation method that reduces the complexity of both the SAM architecture and the prototype classification head while maintaining high-quality semantic segmentation. We evaluate ProtoSAM-2D on multi-organ segmentation tasks across two imaging modalities, demonstrating its effectiveness in zero-shot and few-shot learning scenarios. By combining the flexibility of SAM with prototype-based learning, ProtoSAM-2D offers a novel solution for adaptable semantic segmentation across diverse medical imaging tasks.