Swin Transformer Embedding MSMDFFNet for Road Extraction From Remote Sensing Images

Abstract

Contextual road features and multiscale spatial semantic information play a vital role in road extraction from remote sensing (RS) images. However, accurately modeling these essential features with current convolutional neural network (CNN)-based road extraction algorithms remains challenging, leading to fragmented roads in occluded areas. Inspired by the self-attention mechanism of transformers in natural language processing (NLP), we propose an innovative MSMDFFNet in conjunction with Swin Transformer (SwinMSMDFFNet) for road extraction from RS images. First, the Swin Transformer is embedded as an auxiliary encoder into the MSMDFFNet to incorporate necessary self-attention mechanisms. Meanwhile, a multigranularity sampling (MGS) module is introduced to enhance the computation of self-attention at multiple granularities by the Swin Transformer. This module specifically transforms the feature maps produced by the main encoder into suitable inputs for the auxiliary encoder. Furthermore, to enhance the connections between adjacent local windows in the auxiliary encoder, a cross-directional fusion (CDF) module is designed for feeding the features of the auxiliary encoder back into the main encoder. Extensive experiments conducted on the DeepGlobe and LSRV datasets demonstrate that our proposed SwinMSMDFFNet has significant advantages in extracting road structure, particularly in areas with long-distance occlusions. It surpasses existing methods in pixel-level metrics such as F1 score, intersection over union (IoU), and connectivity metric average path length similarity (APLS). The code will be made publicly available at: https://github.com/wycloveinfall/SwinMSMDFFNet.