Private image synthesis of latent diffusion model with the ciphertext of prompt

Abstract

Artificial intelligence generated content (AIGC) has made significant strides in enabling users to create various realistic visual content, such as images, videos and audio. Diffusion models have shown promise in generating higher-quality images when the user inputs the prompts. However, the prompts and the generated images may contain some privacy information. To address these concerns, this paper proposes a privacy-preserving image synthesis protocol with the ciphertext of prompt. Specifically, we employ differential privacy-stochastic gradient descent (DP-SGD) to update parameters within the Unet instead of the entire parameters, while protecting the privacy of the generated image without decreasing its quality. To ensure prompt confidentiality, we utilize function encryption and design a secure cross-attention for subsequent propagation. Furthermore, based on the theoretical analysis and experimental results, the generated images under the ciphertext prompts can achieve differential privacy, and were almost identical to those under the plaintext prompts.