Weakly Supervised RGBT salient object detection via SAM-Guided Label Optimization and Progressive Cross-modal Cross-scale Fusion

Abstract

Current fully supervised RGB-Thermal salient object detection (RGBT SOD) methods rely on labor-intensive pixel-wise annotations. This work explores weakly supervised RGBT SOD using scribble annotations to reduce the annotation cost. Existing scribble-supervised methods mainly rely on pseudo-labels, which often suffer from issues such as information redundancy, incomplete objects, or inaccurate boundaries. Inspired by the advanced Segment Anything Model (SAM), we propose a two-stage SAM-Guided Label Optimization method to obtain accurate pseudo-labels. The first stage employs the prompt-based segmentation ability of SAM to generate initial masks from RGB and thermal images, while the second stage refines these initial masks using the zero-shot segmentation capability of SAM and complementary information from RGB and thermal modalities. Moreover, existing multi-modal fusion methods might not fully synergize the interactions between channel and spatial dimensions, and they often neglect effective cross-scale feature collaboration during the multi-modal fusion. To address this, we propose the Progressive Cross-modal Cross-scale Fusion Unit (PCCFU), which fuses same-level multi-modal features while progressively integrating higher-level features. PCCFU consists of the Dual Cross-attention Fusion Module and the Cross-scale Aggregation Module. The former facilitates synergistic interactions across both the channel and spatial dimensions, while the latter interacts individually with each higher-level feature during multi-modal feature fusion. Extensive experiments indicate that our method outperforms most fully supervised RGBT SOD approaches and surpasses the previous state-of-the-art weakly supervised method, achieving average improvements of 3.4% in mean F-measure ($F_{\beta }^{avg}$) and 21.5% in mean absolute error ($MAE$) across three benchmark datasets. The code is available at: https://github.com/tzz-ahu.