Parameter efficient face frontalization in image sequences via GAN inversion

Abstract

Processing facial images with varying poses is a significant challenge. Most existing face frontalization methods rely on heavy architectures that struggle with small datasets and produce low-quality images. Additionally, although video frames provide richer information, these methods typically use single images due to the lack of suitable multi-image datasets. To address these issues, a parameter-efficient framework for high-quality face frontalization in both single and multi-frame scenarios is proposed. First, a high-quality, diverse dataset is created for single and multi-image face frontalization tasks. Second, a novel single-image face frontalization method is introduced by combining GAN inversion with transfer learning. This approach reduces the number of trainable parameters by over 91% compared to existing GAN inversion methods while achieving far more photorealistic results than GAN-based methods. Finally, this method is extended to sequences of images, using attention mechanisms to merge information from multiple frames. This multi-frame approach reduces artefacts like eye blinks and improves reconstruction quality. Experiments demonstrate that this single-image method outperforms pSp, a state-of-the-art GAN inversion method, with a 0.15 LPIPS improvement and a 0.10 increase in ID similarity. This multi-frame approach further improves identity preservation to 0.87, showcasing its effectiveness for high-quality frontal-view reconstructions.