An Optimized Transfer Attack Framework Towards Multi-Modal Machine Learning

Deep neural networks (DNNs) have excelled at a wide range of tasks, including computer vision (CV), natural language processing (NLP), and speech recognition. However, past research has demonstrated that DNNs are vulnerable to adversarial examples, which are deliberately meant to trick models into making incorrect predictions by adding subtle perturbations into inputs. Adversarial examples create an exponential threat to multi-modal models that can accept a variety of inputs. By attacking substitute models, we provide a transferable attack framework. The suggested framework optimizes the attack process by modifying the prompt templates and simultaneously raising the attack on multiple inputs. Our experiments demonstrate that the proposed attack framework can significantly improve the success rate of transferable attacks, and adversarial examples are rarely noticed by humans. Meanwhile, experiments show that in transferable attacks, coarse-grained adversarial examples can achieve higher attack success rates than fine-grained ones, and the multi-modal models has some robustness against uni-modal attacks.