Keep and Extent: Unified Knowledge Embedding for Few-Shot Image Generation

Abstract

Training Generative Adversarial Networks (GANs) with few-shot data has been a challenging task, which is prevalently solved by adapting a deep generative model pre-trained on the large-scale data in a source domain to small target domains with limited training data. In practice, most of the existing methods focus on designing task-specific fine-tuning strategies or regularization terms to select and preserve compatible knowledge across the source and target domain. However, the compatible knowledge greatly depends on the target domain and is entangled with the incompatible one. For the few-shot image generation task, without accurate compatible knowledge as prior, the generated images will strongly overfit the scarce target images. From a different perspective, we propose a unified learning paradigm for better knowledge transfer, i.e., keep and extent (KAE). Specifically, we orthogonally decompose the latent space of GANs, where the resting direction that has an unnoticeable impact on the generated images is adopted to extend the new target latent subspace while the remaining directions keep intact to reconstruct the source latent subspace. In this way, the whole source domain knowledge is included in the source latent subspace and the compatible knowledge will be automatically transferred to the target domain along the resting direction, rather than manually selecting. Extensive experimental results on several benchmark datasets demonstrate the superiority of our method.