SET: Superpixel Embedded Transformer for skin lesion segmentation

Abstract

Accurate skin lesion segmentation is crucial for the early detection and treatment of skin cancer. Despite significant advances in deep learning, current segmentation methods often struggle to fully capture global contextual information and maintain the structural integrity of skin lesions. To address these challenges, this paper introduces Superpixel Embedded Transformer (SET), which integrates superpixels into the Transformer framework for skin lesion segmentation. Instead of embedding non-overlapping patches as tokens, SET employs an Association Embedded Merging & Dispatching (AEM&D) module to treat superpixels as the fundamental units during both the down-sampling and up-sampling phases. To better capture the multi-scale information of lesions, we propose a superpixel bank to store various superpixel maps with distinct compactness values. An Ensemble Fusion and Refinery (EFR) module is then designed to fuse and refine the results obtained from each map in the superpixel bank. This approach enables the model to selectively focus on different features by adopting various superpixel maps, thereby enhancing the segmentation performance. Extensive experiments are conducted on multiple skin lesion segmentation datasets, including ISIC 2016, ISIC 2017, and ISIC 2018. Comparative analyses with state-of-the-art methods showcase SET’s superior performance, and ablation studies confirm the effectiveness of our proposed modules incorporating superpixels into Vision Transformer. The source code of our SET will be available at https://github.com/Wzhjerry/SET.