Deep neighbor-coherence hashing with discriminative sample mining for supervised cross-modal retrieval

Abstract

Deep supervised cross-modal hashing has attracted extensive attention because of its low cost and high retrieval efficiency. Although the existing deep supervised cross-modal hashing methods have made great progress, they still suffer from two factors in the preservation of semantic relations between heterogeneous modalities. (1) Most of the available deep supervised cross-modal hashing learn hash functions by employing either pair-wise/multi-wise loss to explore the point-to-point relation or class center loss to explore the point-to-class relation, ignoring collaborative semantic relations. (2) Compared with the large proportion of simple samples, the hard pairs with a small proportion could provide more valuable information for the model training, nevertheless, most deep hash treats all samples equally in the learning process, and overlooks the positive contribution of hard samples in the learning process, impeding the hash function learning. To address these challenges, by considering both point-to-point and point-to-class relations, the novel Deep Neighbor-coherence Hashing (DNcH) framework is proposed to preserve the consistency of neighbor relations and generate high-quality binary codes with intra-class compactness and inter-class separability. Specifically, by jointly exploring the point-to-point and point-to-class relations between heterogeneous data, the neighbor-aware constraint is proposed to project the heterogeneous data into a unified Hamming space, where each anchor is close to all similar samples and corresponding class center, and far away from dissimilar samples and their class centers. The hard pairs containing valuable information are effectively mined by introducing the multi-similarity measurement strategy between heterogeneous modalities to construct the informative and representative training batches. Besides, to further gradually capture discriminant information from multi-modal hard pairs, a self-paced learning mechanism is introduced to assign dynamic weights to multi-modal pairs, which enables the deep cross-modal hashing to gradually concentrate on hard pairs while jointly learning universal patterns from the entire set of multi-modal pairs. Extensive experiments on three benchmark datasets show that our DNcH framework has better performance than the most advanced cross-modal hashing methods. The source code for the DNcH framework is available at https://github.com/QinLab-WFU/DNcH.