Urban region function classification via fusing optical imagery and social media data: A spatio-temporal Transformer interaction approach

Abstract

Urban management and planning can benefit from the classification of urban functional areas. Existing researches have demonstrated that remote sensing data can provide essential urban surface spatial information for the identification of urban functional areas, and human activities may characterize the dynamic aspects connected to social and economic temporal information in various urban functional regions. However, current methods lack explicit consideration of the mutual properties of spatial and temporal modality, resulting in suboptimal interaction performance. To address the issue, we propose a novel fusion method of spatio-temporal Transformer of remote sensing and social media data for urban region function classification. We design the Multi-scale Vision Transformer (MultiViT) to extract the multi-scale features of optical image data and Convolutional Transformer (ConvTransformer) to obtain the multi-scale temporal scale features of social media time series data. For multi-modal fusion, we create a crucial spatio-temporal fusion path based on self-attention, using the different modalities semantic information regarded as useful priori information. By the supervised and distilled loss function, the merging of the two sub-networks and the main network is taken into account during training. Extensive experiments on public datasets have demonstrated the favorable performance of our spatio-temporal Transformer interaction approach in merging remote sensing and social media data for urban region function classification. The code will be available at https://github.com/Ruiyang-Sun/Spatio-temporal-Transformer-for-urfc.