MBT-Polyp: A new Multi-Branch Memory-augmented Transformer for polyp segmentation

Polyp segmentation plays a critical role in the early diagnosis and precise clinical intervention of colorectal cancer (CRC). Despite significant advancements in deep learning for medical image segmentation, accurate localization of small polyps and precise delineation of polyp boundaries remain challenges in colorectal polyp segmentation. In this study, we introduce MBT-Polyp, a Multi-branch Memory-augmented Transformer architecture designed to improve segmentation sensitivity for small polyps and enhance the delineation accuracy of ambiguous polyp boundaries. At the core of our framework is MemoryFormer, a Transformer-based U-shaped architecture that incorporates three key components: a Dynamic Focal Attention block (DFA) for efficient small target enhancement and edge refinement, a High-Level Memory Attention Module (HMAM) for preserving boundary details via cross-resolution fusion, and a Multi-View Channel Memory Attention Module (MCMAM) for suppressing background noise and modeling local spatial context. To guide specialized learning, we derive small polyp and edge labels alongside ground truth, enabling MemoryFormer to process them through dedicated branches. The outputs are fused using a Small Polyp Fusion Strategy (SPFS) and an Edge Correction Strategy (ECS) to alleviate over- and under-segmentation. The quantitative results on Kvasir-SEG, CVC-ColonDB, CVC-ClinicDB, CVC-300, and ETIS-Larib yield mean Dice scores of 0.930, 0.818, 0.943, 0.912, and 0.763, respectively, demonstrating strong generalization across diverse polyp segmentation scenarios. Code and datasets are available at: https://github.com/taojlu/PolypSeg.