Task-specific prompting SAM for multi-task gastric cancer diagnosis in endoscopic images

Abstract

Deep learning is increasingly applied in gastroscopic imaging to assist in lesion detection and diagnosis. However, less attention has been given to analyzing critical lesion characteristics, such as differentiation and invasion depth-factors essential for prognosis and treatment planning. The Segment Anything Model (SAM) introduces robust semantic feature extraction capabilities paired with flexible prompting mechanisms, enabling the model to focus on specific regions of interest. However, designing prompts that effectively capture the diverse semantic information needed for distinct tasks remains underexplored. In this paper, we propose Task-Specific Prompt SAM (TSP-SAM), an innovative framework that employs Task-Specific Prompt Generation (TSPG) and Embedding Prompt Fusion (EPF) to capture multi-perspective features for comprehensive classification and segmentation. Specifically, TSP-SAM utilizes SAM’s powerful feature extraction by integrating multi-scale embeddings and distilled prompt information. This allows the model to refine segmentation, staging, differentiation, and infiltration depth classification within a unified framework. We evaluate TSP-SAM on a newly developed gastroscopy dataset comprising over 3600 images with pixel-level and pathology-based annotations. Extensive experiments demonstrate that TSP-SAM consistently outperforms both traditional and advanced single-task models, showcasing its superior capability for joint optimization across multiple tasks.