A mining scene understanding framework with limited labeled samples jointly driven by object-level spatial relationships and multi-task network

Abstract

Accurately delineating mining areas is crucial for monitoring illegal mining activities. Currently, in large-scale and limited labeled sample scenes, mining areas are easily interfered by targets such as clouds, farmland, and roads with similar spectral characteristics, leading to serious misclassification issues. In response to this problem, this paper proposes a mining area understanding framework based on object-level spatial relation constraints, simulating the way that humans interpret mining scenes in remote sensing images. The framework first constructs a Multi-task Network for joint Panoptic Segmentation and Relation Prediction (PSRP-MNet), aiming to achieve high-precision segmentation of mining area scenes and acquisition of explicit object-level spatial relations. The network contained an explicit spatial relation matching module, a lightweight segmentation head, and multi-scale deformable attention to achieve a comprehensive fusion of deep-level features between different tasks and thus realize a rational utilization of multi-scale semantic information. The spatial relation matching module explicitly models and matches the spatial positional relations between targets existing in mining areas, helping the model understand mining areas from the perspective of object-level targets. The lightweight design of the segmentation head maintains high performance while reducing model complexity and parameters. Subsequently, the spatial relations were matched with the prior object-level spatial relation knowledge criteria constructed in this paper, determining the integrated functional structures in the scene to further constrain the segmentation results. The guidance of spatial relations helps PSRP-MNet correct its predictions when errors occur, leading to excellent performance in limited labeled sample tasks. Two sufficiently large scenes were selected as study areas, and approximately 1000 image samples were used for training. Multiple sets of comparative experiments were conducted to validate the framework’s effectiveness and cross-regional generalization ability in limited labeled sample tasks. It was observed that the introduction of spatial relations and the association between different tasks reduced the error accumulation of PSRP-MNet. This research is expected to provide a reference for the regular monitoring of mineral resources.