A landmarks-assisted diffusion model with heatmap-guided denoising loss for high-fidelity and controllable facial image generation

Diffusion models have significantly advanced image generation, enabling users to create diverse and realistic images from simple prompts. However, generating high-fidelity, controllable facial images remains a challenge due to the intricate details of human faces. In this paper, we present a novel diffusion model for landmarks-assisted text to face generation which directly incorporates landmarks as guidance during the diffusion process. To address the issue of global information degradation caused by fine-tuning with local information, we introduce a heatmap-guided denoising loss that selectively focuses on feature pixels most relevant to the conditioning. This biased learning strategy ensures that the model prioritizes shape and positional information, preventing excessive deterioration of its generalization ability. Unlike existing methods relying on an extra learnable branch for conditional control, our native method eliminates the conflicts inherent in dual-branch architectures when dealing with various conditions. It also enables precise manipulation of facial features, such as shape and position. Extensive experiments on CelebA-HQ and CelebAText-HQ dataset show that our method demonstrates superior performance in generating realistic and controllable facial images, outperforming existing methods in terms of fidelity, diversity, and alignment with specified landmarks.