Multi-E2E: An end-to-end urban land-use mapping framework integrating high-resolution remote sensing images and multi-source geographical data

The urban land-use map reflects the distribution of the different functional lands in the city, serving as a valuable reference for urban planning and management. High-resolution remote sensing (HRS) images provide detailed spatial information about parcels but lack socio-economic information, which is crucial for identifying land-use types. To enhance the mapping performance and obtain more comprehensive land-use information, the integration of HRS images and points of interest (POIs) with socio-economic information is crucial. Nonetheless, the existing land-use mapping methods based on HRS images and POIs are generally confronted with the following challenges: 1) due to the reliance on prior knowledge, the existing methods cannot automatically capture the complex relationships between multi-source data and land-use categories; 2) there are substantial semantic disparities between HRS images and POIs, so that the simple fusion approaches cannot fully utilize the complementary information; and 3) the existing methods are generally based on the assumption of complete modalities, resulting in them failing to work on POI-deficient parcels. In this paper, to address these issues, an end-to-end urban land-use mapping framework integrating HRS images and multi-source geographic data (Multi-E2E) is proposed. The Multi-E2E framework automatically establishes the mapping from multi-source data to land-use categories through a data-driven approach, and generates informative fused representations with an adaptive fusion module (AFM). In Multi-E2E, the labeled HRS image-POI pairs are constructed using the areas of interest (AOIs), and the interactions between modalities are facilitated by the end-to-end architecture. To identify POI-deficient parcels and ensure that the modality-specific encoders are adequately supervised, a unimodal supervision module (USM) is introduced in the Multi-E2E framework. Experiments conducted with multi-source samples from 34 Chinese cities and the urban regions of Beijing, Wuhan, Hong Kong, Macao, and Helsinki validate the effectiveness and generalizability of the proposed framework for urban land-use mapping applications. The code will be publicly available at https://github.com/Rayoll/Multi_E2E.