DiffInpaint: line drawing guided murals restoration with diffusion model

The artistic value of murals is immensely precious, yet these murals are highly susceptible to damage from natural or anthropogenic factors. Mural images present challenges such as semantic loss and texture ambiguity, which limit the effectiveness of conventional deep learning methods. This study aims to develop an effective digital restoration framework addressing complex structural and textural issues characteristic of mural images. To achieve this, we propose a mural restoration method based on a diffusion model. The approach involves simulating damaged mural regions by repeatedly adding Gaussian noise during the forward process of the diffusion model. In the reverse generation phase, lines are utilized as conditional inputs to guide and enhance the U-net network’s structural and textural predictions. Additionally, a two-stage training strategy is introduced: first, a line encoder is pre-trained to generate conditional feature maps for the Latent Diffusion Model (LDM); subsequently, the LDM is trained based on these conditioned maps. Experimental results indicate that our method effectively repairs various mural defects and damages, maintaining overall stylistic consistency and detail. This approach contributes to the quality and efficiency of cultural heritage image restoration, providing a viable technical support for digital mural preservation and restoration.