Automated prompt-guided multi-modality cell segmentation with shape-aware classification and boundary-aware SAM adaptation

Abstract

Robust and accurate cell segmentation across diverse imaging modalities remains a critical challenge in microscopy image analysis. While foundation models like the Segment Anything Model (SAM) have demonstrated exceptional performance in natural image segmentation, their adaptation to multi-modal cellular analysis is hindered by domain-specific knowledge gaps and morphological complexity. To bridge this gap, we present a novel SAM-driven framework featuring three systematic innovations: First, we propose Shape-Aware Classification to enhance segmentation of cells with diverse morphologies. Second, Auto Point Prompt Generation (APPGen) module guides the segmentation model with automatically generated point cues to improve segmentation accuracy. Third, we implement Boundary-Aware SAM Adaptation to effectively resolve overlapping cells in microscopy images. Our experiments show that the proposed framework reduces manual effort through automated prompts, adapts well to different imaging modalities, and enhances segmentation accuracy by incorporating boundary-aware techniques. The source code is available at https://github.com/MIXAILAB/Multi_Modality_CellSeg.