CRFFNet: A cross-view reprojection based feature fusion network for fine-grained building segmentation using satellite-view and street-view data

Abstract

Fine-grained building attribute segmentation is crucial for rapidly acquiring urban geographic information and understanding urban development dynamics. To achieve a comprehensive perception of buildings, fusing cross-view data, which combines the wide coverage of satellite-view imagery with the detailed observations of street-view images, has become increasingly important. However, existing methods still struggle to effectively mitigate feature discrepancies across different views during cross-view fusion. To address this challenge, we propose the CRFFNet, a Cross-view Reprojection-based Feature Fusion Network for fine-grained building attribute segmentation. CRFFNet eliminates the perspective differences between satellite-view (satellite image and map data) and street-view features, enabling high-precision building attribute segmentation. Specifically, we introduce a deformable module to reduce target distortions in panoramic street-view images, and develop an Explicit Geometric Reprojection (EGR) module, which leverages explicit BEV geometric priors to reproject street-view features onto the satellite-view plane without requiring complex parameter inputs or depth information. To support evaluation, we construct two new datasets, Washington and Seattle, which include satellite imagery, map data, and panoramic street-view images, serving as benchmarks for cross-view, fine-grained building attribute segmentation. Extensive experiments conducted on these datasets, as well as on the public OmniCity and Brooklyn datasets, demonstrate that CRFFNet achieves mIoU improvements of 1.02% on Washington, 8.12% on Seattle, 2.29% on OmniCity, and 2.87% on Brooklyn compared to the second-best method. These improvements demonstrate the potential of our CRFFNet for applications involving large-scale multi-source data, contributing to more comprehensive urban analysis and planning.