Harmony score-guided inpainting: Iterative refinement for seamless image inpainting

Abstract

Inpainting techniques often demand extensive model fine-tuning or the concatenation of latent vectors, which can be time-intensive and prone to overfitting. Such methods frequently lead to inconsistencies between the inpainted regions and the surrounding background, occasionally producing partially satisfactory results where some areas appear natural while others remain unrealistic. To address these limitations, we demonstrate that existing inpainting methods can sufficiently handle certain scenarios but may struggle with specific problematic patches. We propose an iterative enhancement approach guided by an Inpainting Harmony Score, which evaluates the coherence of the inpainted image. Our method selectively enhances only the poorly reconstructed patches, preserving their masks for subsequent inpainting iterations. The process is repeated, followed by a final blending step to ensure seamless integration between the inpainted region and the background. This approach improves the overall quality and consistency of inpainting results while minimizing the risks of overfitting and inefficiency.