Fooled by Imagination: Adversarial Attack to Image Captioning Via Perturbation in Complex Domain

Shaofeng Zhang, Zheng Wang, Xing Xu, Xiang Guan, Yang Yang
{"title":"Fooled by Imagination: Adversarial Attack to Image Captioning Via Perturbation in Complex Domain","authors":"Shaofeng Zhang, Zheng Wang, Xing Xu, Xiang Guan, Yang Yang","doi":"10.1109/icme46284.2020.9102842","DOIUrl":null,"url":null,"abstract":"Adversarial attacks are very successful on image classification, but there are few researches on vision-language systems, such as image captioning. In this paper, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises in complex domain. In particular, we propose Fooled-by-Imagination, a novel algorithm for crafting adversarial examples with semantic embedding of targeted caption as perturbation in complex domain. The proposed algorithm explores the great merit of complex values in introducing imaginary part for modeling adversarial perturbation, and maintains the similarity of the image in real part. Our approach provides two evaluation approaches, which check whether neural image captioning systems can be fooled to output some randomly chosen captions or keywords. Besides, our method has good transferability under black-box setting. At last, our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords at a higher success rate.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"11","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2020 IEEE International Conference on Multimedia and Expo (ICME)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/icme46284.2020.9102842","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 11

Abstract

Adversarial attacks are very successful on image classification, but there are few researches on vision-language systems, such as image captioning. In this paper, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises in complex domain. In particular, we propose Fooled-by-Imagination, a novel algorithm for crafting adversarial examples with semantic embedding of targeted caption as perturbation in complex domain. The proposed algorithm explores the great merit of complex values in introducing imaginary part for modeling adversarial perturbation, and maintains the similarity of the image in real part. Our approach provides two evaluation approaches, which check whether neural image captioning systems can be fooled to output some randomly chosen captions or keywords. Besides, our method has good transferability under black-box setting. At last, our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords at a higher success rate.
被想象愚弄:复域扰动对图像标题的对抗性攻击
对抗性攻击在图像分类方面非常成功,但在图像字幕等视觉语言系统方面的研究还很少。本文研究了一种基于CNN+RNN的图像字幕系统在复杂域对抗噪声下的鲁棒性。特别地,我们提出了一种新的算法——用目标标题的语义嵌入作为复杂域的扰动来制作对抗示例。该算法充分利用复值在引入虚部建模对抗摄动方面的优点,并在实部保持图像的相似性。我们的方法提供了两种评估方法,用于检查神经图像字幕系统是否可以被愚弄以输出一些随机选择的字幕或关键词。此外,该方法在黑箱设置下具有良好的可移植性。最后,我们的大量实验表明,我们的算法可以以更高的成功率成功地制作具有随机目标标题或关键词的视觉相似的对抗示例。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书