Text optimization with latent inversion for non-rigid image editing

Abstract

Text-guided non-rigid image editing involves complex edits for input images, such as changing motion or compositions of the object (e.g., making a horse jump or adding candles on a cake). Since it requires manipulating the structure of the object, existing methods often compromise “image identity”– defined as the overall object appearance and background details – particularly when combined with Stable Diffusion. In this work, we propose a new approach for non-rigid image editing with Stable Diffusion, aimed at improving the image identity preservation quality without compromising editability. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. Inspired by the success of Imagic, we employ their text optimization for smooth editing. Then, we introduce latent inversion to preserve the input image’s identity without additional model fine-tuning. To fully utilize the input reconstruction ability of latent inversion, we employ timestep-aware text injection sampling, strategically injecting the source text prompt in early sampling steps and then transitioning to the target prompt in subsequent sampling steps. This strategic approach seamlessly harmonizes with text optimization, facilitating complex non-rigid edits to the input without losing the original identity. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality through extensive experiments. Our code is available at https://github.com/YunjiJung0105/TOLI-non-rigid-editing.