FaceChain-MMID: Generating highly identity-consistent realistic portraits via dividing & merging multi-modal representations

Recent advancements in text-to-image generation have made significant strides in customizing realistic human photos. Most of the existing methods focus on addressing efficiency issues to avoid resource-intensive and time-consuming subject-specific fine-tuning. However, they lack an in-depth exploration for identity preservation, thus suffering from significant degradation in real scenarios. We propose FaceChain-MMID in response to this challenge. First, we comprehensively represent facial identity using three factors: the face image to provide basic identity, the segmentation mask to refine the facial geometry, the text prompts to further supplement additional identity-related attributes. Building upon these multi-modal features, we propose a novel dividing and merging strategy to support highly identity-consistent personalized portrait generation. Specifically, the dividing stage ensures that each modality fully expresses its own information by training independent uni-modal conditional diffusion. The subsequent merging stage introduces an efficient modal-specific proxy module to sufficiently combine the noise from each branch at latent denoising steps, where incorporates identity-centric face pairs, a filtering mechanism, and truncated loss to enhance inter-modal complementarity. Extensive qualitative and quantitative experiments demonstrate the superior performance of our approach in preserving identity.