Mutual Quantization for Cross-Modal Search with Noisy Labels

Erkun Yang, Dongren Yao, Tongliang Liu, Cheng Deng
{"title":"Mutual Quantization for Cross-Modal Search with Noisy Labels","authors":"Erkun Yang, Dongren Yao, Tongliang Liu, Cheng Deng","doi":"10.1109/CVPR52688.2022.00740","DOIUrl":null,"url":null,"abstract":"Deep cross-modal hashing has become an essential tool for supervised multimodal search. These models tend to be optimized with large, curated multimodal datasets, where most labels have been manually verified. Unfortunately, in many scenarios, such accurate labeling may not be avail-able. In contrast, datasets with low-quality annotations may be acquired, which inevitably introduce numerous mis-takes or label noise and therefore degrade the search per-formance. To address the challenge, we present a general robust cross-modal hashing framework to correlate distinct modalities and combat noisy labels simultaneously. More specifically, we propose a proxy-based contrastive (PC) loss to mitigate the gap between different modalities and train networks for different modalities Jointly with small-loss samples that are selected with the PC loss and a mu-tual quantization loss. The small-loss sample selection from such Joint loss can help choose confident examples to guide the model training, and the mutual quantization loss can maximize the agreement between different modalities and is beneficial to improve the effectiveness of sample selection. Experiments on three widely-used multimodal datasets show that our method significantly outperforms existing state-of-the-arts.","PeriodicalId":355552,"journal":{"name":"2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"7","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/CVPR52688.2022.00740","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 7

Abstract

Deep cross-modal hashing has become an essential tool for supervised multimodal search. These models tend to be optimized with large, curated multimodal datasets, where most labels have been manually verified. Unfortunately, in many scenarios, such accurate labeling may not be avail-able. In contrast, datasets with low-quality annotations may be acquired, which inevitably introduce numerous mis-takes or label noise and therefore degrade the search per-formance. To address the challenge, we present a general robust cross-modal hashing framework to correlate distinct modalities and combat noisy labels simultaneously. More specifically, we propose a proxy-based contrastive (PC) loss to mitigate the gap between different modalities and train networks for different modalities Jointly with small-loss samples that are selected with the PC loss and a mu-tual quantization loss. The small-loss sample selection from such Joint loss can help choose confident examples to guide the model training, and the mutual quantization loss can maximize the agreement between different modalities and is beneficial to improve the effectiveness of sample selection. Experiments on three widely-used multimodal datasets show that our method significantly outperforms existing state-of-the-arts.
带噪声标签的跨模态搜索的相互量化
深度跨模态哈希已经成为监督式多模态搜索的重要工具。这些模型倾向于使用大型、精心策划的多模态数据集进行优化,其中大多数标签都是手动验证的。不幸的是,在许多情况下,可能无法获得这种准确的标签。相反,可能会获得具有低质量注释的数据集,这不可避免地会引入许多错误或标签噪声,从而降低搜索性能。为了应对挑战,我们提出了一个通用的鲁棒跨模态哈希框架,以关联不同的模态并同时对抗噪声标签。更具体地说,我们提出了一种基于代理的对比(PC)损耗,以减轻不同模式之间的差距,并与使用PC损耗和相互量化损耗选择的小损耗样本联合训练不同模式的网络。从这种联合损失中选择小损失样本可以帮助选择自信样本来指导模型训练,相互量化损失可以最大限度地提高不同模式之间的一致性,有利于提高样本选择的有效性。在三个广泛使用的多模态数据集上的实验表明,我们的方法明显优于现有的最先进的方法。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信