StableID: Multimodal learning for stable identity in personalized Text-to-Face generation

Abstract

Personalized Text-To-Face (TTF) generation aims to inject new subjects (e.g., identity information) into the text-to-image diffusion model, generating images that align with text prompts and maintain subject consistency in different contexts. Currently, some methods usually overfit the reference images on text prompts related to facial attributes, or ignore facial details to fit the text prompts, thus weakening identity consistency. To address these issues, we propose a personalized TTF method for generating Stable IDentities without fine-tuning, named StableID. Firstly, multimodal-guided identity constraint is proposed to ensure stability of identity features and preservation of face details, along with semantic editing capabilities. Secondly, we design residual cross-attention based mask balancing loss that effectively separates identity information from non-identity related backgrounds, balancing the effects of text prompts and identity constraints. Furthermore, we develop a portrait dataset with detailed facial prompt, as well as decoupled editable attribute vectors, enabling smooth and precise control over fine-grained semantic edits. Extensive experimental results show that our method outperforms the state-of-the-arts in stable identity consistency.