TPA-Seg: Multi-Class Nucleus Segmentation Using Text Prompts and Cross-Attention

Abstract

Precise semantic segmentation of nuclei in pathological images is a crucial step in pathological diagnosis and analysis. Given the limited scale and the high cost of annotation for current pathological datasets, appropriately incorporating textual prompts as prior knowledge is key to achieving high-accuracy multi-class segmentation. These text prompts can be derived from image information such as the morphology, size, location, and density of nuclei in medical images. The text prompts are processed by a text encoder to obtain textual features, while the images are processed by an image encoder to obtain multi-scale feature maps. These features are then fused through feature fusion blocks, allowing the features to interact and be perceived in a multi-scale multimodal manner. Finally, metric learning and weighted loss functions are introduced to prevent feature loss caused by a small number of categories or small target sizes in the image. Experimental results on multiple pathological image datasets demonstrate that our method is effective and outperforms existing models in the segmentation of pathological images. Furthermore, the study verifies the effectiveness of each module and evaluates the potential of different types of text prompts in improving performance. The insights and methods proposed may offer a novel solution for segmentation and classification tasks. The code can be viewed at https://github.com/kahhh743/TPA-Seg.