FeaST: Feature-guided Style Transfer for high-fidelity art synthesis

Text-conditioned image synthesis methods such as DALLE-2, IMAGEN, and Stable Diffusion are gaining strong attention from deep learning and art communities recently. Meanwhile, Image-to-Image (Img2Img) synthesis applications that emerged from the pioneering Neural Style Transfer (NST) approach have swiftly transitioned towards the feed-forward Automatic Style Transfer (AST) methods, due to numerous constraints inherent in the former method, including inconsistent synthesis outcomes and sluggish optimization-based synthesis process. However, NST holds significant potential yet remains relatively underexplored within this research domain. In this paper, we revisited the original NST method and uncovered its potential to attain image quality comparable to the AST synthesis methods across a diverse range of artistic styles. We propose a two-stage Feature-guided Style Transfer (FeaST) which consists (a) pre-stylization step called Sketching to address the poor initialization issue, and (b) Finetuning to guide the synthesis process based on high-frequency (HF) and low-frequency (LF) guidance channels. By addressing the issues of inconsistent synthesis and slow convergence inherent in the original method, FeaST unlocks the full capabilities of NST and significantly enhances its efficiency.