SMRFHash: Semantic-Guided Multi-Representation Fusion Hashing for Fine-Grained Image Retrieval

Abstract

Fine-Grained Image Retrieval (FGIR) is an important visual task. Most existing hash-based FGIR methods have difficulty in sufficiently leveraging low-level attributes and effectively associating high-level semantic information with these low-level attributes, thereby limiting their ability to accurately discriminate fine-grained images. To address this issue, we propose a Semantic-Guided Multi-Representation Fusion Hashing (hereafter SMRFHash), which uses semantic features as an anchor to guide and align color and frequency features, thereby enhancing fine-grained retrieval. Specifically, a multi-branch feature extraction module is proposed to separately capture frequency, color, and semantic features of images by attention mechanisms. Then, we further develop a semantic-guided attention fusion module that guides the model to focus on frequency and color regions relevant to the target object. In addition, we propose a confidence-weighted hash loss, which jointly exploits the triplet loss and cross-entropy loss to constrain network training. The proposed loss dynamically re-weights training samples according to the model’s prediction confidence. It improves early-stage training stability and enhances discriminative learning from hard samples. Experimental results show that the SMRFHash outperforms several state-of-the-art methods in retrieval performance.