Attacking DNN-based Cross-modal Retrieval Hashing Framework with Adversarial Perturbations

The rapid development of Internet and online data explosions elicit strong aspirations for users to search for semantic relevant information based on available samples. While the data online are always released with different modalities like images, videos or texts, effective retrieval models should discover latent semantic information with different structures. Recently, the state-of-the-art deep cross modal retrieval frameworks have effectively enhanced the performance on commonly-used platforms using the deep neural networks (DNNs). Yet DNNs have been verified to be easily misguided by small perturbations, and there are already several attack generation methods proposed on DNN-based models for real-world tasks, but they are all focused on supervised tasks like classification or object recognition. To effectively evaluate the robustness of deep cross-modal retrieval frameworks, in this paper, we propose a retrieval-based adversarial perturbation generation method, and demonstrate that our perturbation could effectively attack the state-of-the-art deep cross-modal and single image retrieval hashing models.