CTD-inpainting: Towards the Coherence of Text-driven Inpainting with Blended Diffusion

Abstract

Text-driven inpainting has emerged as a prominent and challenging research topic in image completion recently, where denoising diffusion probabilistic models (DDPM)-based approaches have achieved state-of-the-art performance on authentic and diverse images. However, ensuring high image fidelity during generation remains a critical aspect in effective text-driven inpainting. Moreover, guaranteeing coherence between the unmasked region (background) and the generated results in the masked regions poses a significant challenge in measurement and implementation. To address these issues, we propose CTD-Inpainting, a novel text-driven inpainting framework, incorporates a coherence constraint between the masked and unmasked regions. Specifically, CTD-Inpainting employs a pre-trained contrastive language-image model (CLIP) to guide DDPM-based generation, aligning it with the text prompt. Additionally, we introduce a transition region between the background and the masked region via mask expansion. This transition region helps maintain coherence between the foreground and background by ensuring consistency between the generated results and the original background during inpainting. At each denoising step, we employ a blending technique, where multiple noise-injected versions of the input image are harmonized with the latent diffusion guided by text and coherence constraint in the transition region. This enables seamless integration of conditional information with the generated information via resampling. Additionally, we design an innovative coherence metric based on the coherence constraint, providing a quantitative measure for the subjective coherence assessment. Extensive experiments manifest the superiority of CTD-Inpainting against state-of-the-art methods on real-world and diverse images.