SAHs: Unraveling Safety Attention Heads in Large Vision-Language Models

Abstract

With the integration of an additional modality, large vision-language models (LVLMs) exhibit greater vulnerability to safety risks (e.g., jailbreaking) compared to their language-only predecessors. Although recent studies have devoted considerable effort to the post-hoc alignment of LVLMs, the inner safety mechanisms remain largely unexplored. In this paper, we discover that activations of LVLM attention heads can pose significant linear separability of malicious and benign prompts. We hypothesize that the model’s inherent safety perception is governed by sparse attention heads, which we term “safety heads.” Further analysis reveals that these heads act as specialized shields against malicious prompts; ablating them leads to higher attack success rates, while the model’s utility remains unaffected, which further confirms our hypothesis. Beyond empirical evidence, we also conduct a semantic analysis showing that safety heads are capable of isolating and amplifying unsafe semantics in both image and text inputs, offering interpretable insights into their protective role. By locating these safety heads and concatenating their activations, we construct a straightforward but powerful malicious prompt detector that integrates seamlessly into the generation process with minimal extra inference overhead. Despite its simple structure of a logistic regression model, the detector surprisingly exhibits strong zero-shot generalization capabilities. Experiments across various prompt-based attacks in both image and video domains confirm the effectiveness of leveraging safety heads to protect LVLMs.