NeSF-Net: Building roof and facade segmentation based on neighborhood relationship awareness and scale-frequency modulation network for high-resolution remote sensing images

Abstract

Building information extraction holds significant application value in smart city development, urban planning, and management. With the accelerating process of urbanization, mid- and high-rise buildings are increasingly prevalent. In orthophotos, the roofs of tall buildings often do not fully overlap with their footprints. In satellite images from oblique angles, buildings may also be obstructed or affected by shadows. Therefore, building information extraction should evolve from a roof-only extraction task to a comprehensive task that includes both roofs and facades. Current methods predominantly employ convolutional neural networks (CNNs) and Transformer models, focusing on describing building boundary and global features. However, these methods have the following limitations: insufficient utilization of information between pixels and limited spatial information recovery capabilities in decoders. This makes it difficult to distinguish between roofs and facades, and the morphological structure of buildings is challenging to maintain. To address these issues, this paper proposes a new network architecture—NeSF-Net, designed to focus on the accurate extraction of roofs and facades. NeSF-Net consists of two core modules: the neighborhood relationship awareness module (NRAM) and the scale-frequency modulation decoder (SFMD). NRAM enhances the connectivity between pixels by constructing sub-neighborhood relationship awareness in the latent space of deep features, effectively improving the integrity of the segmentation results. SFMD significantly reduces the loss of spatial information during the upsampling process by thoroughly extracting and integrating the scale and frequency features of buildings in the decoder. Experiments were conducted on the BANDON dataset, which contains images captured from oblique angles. The proposed method achieved a mIoU of 72.71 % and an F1 score of 83.04 %, outperforming state-of-the-art segmentation methods. The performance in facade extraction was particularly notable, with a mIoU score exceeding the second-best method by 4.92 %. Additionally, generalization experiments were conducted using GaoFen-7 satellite images, taking Shenzhen as a case study. The results demonstrate that the proposed method exhibits good generalization and robustness.