Dual aggregation based joint-modal similarity hashing for cross-modal retrieval

Abstract

Cross-modal hashing aims to leverage hashing functions to map multimodal data into a unified low-dimensional space, realizing efficient cross-modal retrieval. In particular, unsupervised cross-modal hashing methods attract significant attention for not needing external label information. However, in the field of unsupervised cross-modal hashing, there are several pressing issues to address: (1) how to facilitate semantic alignment between modalities, and (2) how to effectively capture the intrinsic relationships between data, thereby constructing a more reliable affinity matrix to assist in the learning of hash codes. In this paper, Dual Aggregation-Based Joint-modal Similarity Hashing (DAJSH) is proposed to overcome these challenges. To enhance cross-modal semantic alignment, we employ a Transformer encoder to fuse image and text features and introduce a contrastive loss to optimize cross-modal consistency. Additionally, for constructing a more reliable affinity matrix to assist hash code learning, we propose a dual-aggregation affinity matrix construction scheme. This scheme integrates intra-modal cosine similarity and Euclidean distance while incorporating cross-modal similarity, thereby maximally preserving cross-modal semantic information. Experimental results demonstrate that our method achieves performance improvements of 1.9 % $\sim$ 5.1 %, 0.9 % $\sim$ 5.8 % and 0.6 % $\sim$ 2.6 % over state-of-the-art approaches on the MIR Flickr, NUS-WIDE and MS COCO benchmark datasets, respectively.