Coarse-to-fine text injecting for realistic image super-resolution

Abstract

Image Super-Resolution (ISR) aims at enhancing the resolution of a given image by guessing the RGB values of additional pixels from existing ones. Equipping with the pre-trained text-to-image diffusion models, i.e. Stable Diffusion (SD), and fine-tuning it for image-to-image purposes, recent ISR methods are able to leverage powerful prior knowledge to generate abundant details from the low-resolution observations. They abolished the capability of using precise text to control image generation and thus might lose the prior can be queried via text for ISR. In this work, we propose a plug-and-play text-injecting strategy that capitalizes on the inherent benefits of text guidance from a pre-trained SD model. Specifically, two levels of text prompts, including general and detailed descriptions, are automatically generated via existing large language models. By introducing a time-aware text injector, we are able to inject text features with varying granularity to generate progressively detailed results as diffusion unfolds. Based on the observation that the stochastic nature of the diffusion model may lead to different reconstruction results, we develop two test-time methods that conduct multiple runs with different initial noises to refine the fidelity or realism of reconstructed images. Additionally, we explore the interpretability of our method by visualizing the cross-attention maps in the denoising U-Net, which show an obvious correspondence between semantically meaningful words in the text prompt and the corresponding image features. Extensive experiments demonstrated the superiority of our method in balancing fidelity and realism over current state-of-the-art approaches.