Mitigating Replication and Bias in Diffusion-Based Face Image Synthesis Models With Rewards

Abstract

Diffusion-based image synthesis models have gained a lot of traction in recent years. However, a notable observation is that when trained on smaller datasets, diffusion models tend to replicate training data. This issue is particularly problematic in domains such as medical imaging and biometrics, where data is both limited and sensitive. Moreover, diffusion models have been observed to amplify facial attribute biases. Therefore, addressing these issues is crucial, as they can have adverse effects on the downstream applications utilizing these diffusion models. In this paper, we propose a novel methodology to mitigate both data replication and social biases by using reward maximization to fine-tune pre-trained unconditional diffusion models, aiming to achieve a desired output distribution. Our results show the effectiveness of the proposed method in successfully mitigating gender and racial biases present in pre-trained unconditional diffusion models employed for face generation. Furthermore, our method significantly reduces data replication in unconditional diffusion models, highlighting its potential to enhance the behavior of these generative models.