Hypergraph-based CLIP hashing for unsupervised cross-modal retrieval

Abstract

With the surge of multi-modal data, how to effectively and efficiently find similar information has become an urgent and important need. Among the existing solutions, unsupervised cross-modal hashing can learn from unlabeled data and provide fast and satisfactory retrieval performance, making it a viable solution. However, existing unsupervised cross-modal hashing methods often inadequately model intricate cross-modal semantic relationships. To bridge this gap, this paper proposes a novel Hypergraph-based CLIP Hashing (HCH). Specifically, HCH utilizes the large-scale visual-language pre-trained model CLIP to extract visual and textual features, and employs a cross-modal Transformer to further enhance semantic fusion among these features. Then, to fully capture the semantic relevance among multi-modal data, we construct a semantic-enhanced similarity matrix and design a mean-based weighting scheme to adjust this matrix. Additionally, we compose a hypergraph convolutional network to further explore high-order semantic information within the input data, leading to more compact and high-quality hash codes. To substantiate HCH’s efficacy, we conducted experiments on three commonly used datasets, confirming its superiority over leading baselines.