FCAT-Diff: Flexible and Consistent Appearance Transfer Based on Training-free Diffusion Model

Abstract

The core goal of appearance transfer is to seamlessly integrate the appearance of a reference image into a content image. However, existing methods operate on the entire image and fail to accurately identify the regions of interest for appearance transfer, leading to structural loss and incorrect background transfer. Additionally, these methods lack flexibility, making it difficult to achieve fine-grained control at the regional level. To address these issues, we propose FCAT-Diff, a training-free framework for flexible and consistent appearance transfer without additional training or fine-tuning. Specifically, to achieve more consistent appearance transfer, we employ a dual-guidance branch to provide structure and appearance features, which are fused through an enhanced self-attention module called Mask-Appearance-Attention (MAA). The MAA clearly distinguishes the boundaries between the background and the transferred region, ensuring consistency in both the structure and background. To increase the flexibility of transfer, we utilize a mask that allows users to select the regions of interest for transfer, enabling appearance transfer for specified regions. Furthermore, given multiple reference images and their corresponding regions, our FCAT-Diff supports the transfer of multiple appearances. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance in maintaining the structural and background consistency of the content image while providing greater flexibility.