A novel knowledge-based multi-modal semi-supervised framework for 3D change detection and mining volume estimation in open-pit mines using GF7 satellite images

Abstract

The detection of three-dimensional (3D) terrain changes in open-pit mines is of great significance for resource management and environmental monitoring. Obtaining multi-temporal, high-quality, large-scale elevation data is very difficult, and the data used in previous studies were insufficient to support large-scale 3D change detection in mining areas. The emergence of the GF7 satellite image has resolved this issue. This paper proposes a framework for 3D change detection in large-scale, few-shot mining areas using GF7 satellite images, which simultaneously outputs the mine’s two-dimensional (2D) and 3D change detection results. From this, it can further estimate the mining volume in the mine. The framework builds a remote sensing knowledge-based multi-modal, semi-supervised mine recognition model, which fuses complementary multi-modal information of the mine from the input DSM and imagery through feature alignment and cross-modal attention mechanisms. It also employs a strong–weak consistency regularization strategy, which integrates spectral and terrain knowledge from unlabeled data to learn the feature differences between the mine and background elements and the heterogeneity of boundaries, thereby enhancing the model’s sensitivity to mine-specific features. The model’s pre- and post-temporal mine predictions are differenced and overlaid with DSM to obtain 2D and 3D change detection results. Based on this, the mining volume is estimated using the difference integration method. Multiple comparison and ablation experiments validate the accuracy of the 2D and 3D change detection, as well as its robustness in dealing with different seasonal change scenarios and severely imbalanced class distributions. The study is expected to provide a reference for monitoring the mining progress of mineral resources. The code of the HD-Net will be made available freely at https://github.com/dongdhcugb/KMS-RNet.git.