PVT-MSFF: A Pyramid Vision Transformer with Multi-Scale Feature Fusion for polyp segmentation in endoscopic images

Abstract

Colorectal cancer (CRC) is one of the most fatal malignancies worldwide, and its early diagnosis relies on the accurate detection and segmentation of polyps in endoscopic images. However, existing methods are often challenged by the diverse morphology of polyps, ambiguous boundaries, and differences in data centers, which can lead to missed detections and limited generalization. Here, we propose a novel Pyramid Vision Transformer with Multi-Scale Feature Fusion network (PVT-MSFF), which combines a Pyramid Vision Transformer encoder with a cooperative multi-module strategy. Our method introduces a Feature Enhancement Module (FEM) with cross-attention mechanism, a Multi-Scale Fusion Module (MSFM) for hierarchical feature expression, and a Global Context Sensing (GCS) module to enhance boundary sensitivity and semantic integration. We evaluate PVT-MSFF on five public polyp segmentation benchmark datasets, including Kvasir-SEG, ClinicDB, ColonDB, ETIS, and CVC-300. Experimental results demonstrate that our method achieves highly competitive segmentation performance, with mDice scores of 0.921, 0.949, 0.813, 0.809, and 0.878 on the respective datasets. Although SAM2-UNet attains 0.928 mDice on Kvasir-SEG and ASPS reaches 0.950 mDice on CVC-ClinicDB, showing results closely comparable to ours, our model exhibits significant advantages in computational efficiency, with parameter count (25.16M) and computational cost (10.12 GFLOPs) substantially lower than these large-scale model-based approaches. This excellent balance between high accuracy and efficiency makes PVT-MSFF particularly valuable for practical clinical applications. Moreover, validation on a clinical dataset (G-endoscope) further confirms robust generalization and accuracy, highlighting the potential of PVT-MSFF for intelligent endoscopy systems and computer-assisted diagnosis. Our code will be available at: https://github.com/jize123457/PVT-MSFF.