EASA: Fine-Tuning SAM with Edge Attention and Adapters for Image Manipulation Localization

Abstract

The Segment Anything Model (SAM) is gaining attention for various applications due to its success in precise segmentation and zero-shot inference across diverse datasets. The image manipulation localization (IML) task, facing a lack of high-quality, diverse datasets, could benefit from SAM's strong generalization ability. However, the unique nature of the IML task presents a significant challenge: the vast distributional disparity between IML data and conventional visual task data makes it seem implausible to effectively transfer a pretrained model like SAM to this task. Models typically either forget previous knowledge or fail to adapt to IML's unique data distribution due to structural mismatches. To address this, we introduce the edge-attention SAM-adapter (EASA), which overcomes catastrophic forgetting and effectively adapts to IML's unique data distribution. Specifically, our EASA method mitigates the issue of catastrophic forgetting by employing adapter tuning strategy and designs a novel edge-attention branch effectively captures the subtle traces of edge manipulations in manipulated regions. In our experiments across six public datasets, our method significantly enhances performance in IML tasks compared to state-of-the-art methods, thus showcasing the potential of SAM in various downstream tasks previously considered challenging.