Multi3D: 3D-aware multimodal image synthesis

IF 17.3 3区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Computational Visual Media Pub Date : 2024-04-03 DOI:10.1007/s41095-024-0422-4

引用次数: 0

Abstract

3D-aware image synthesis has attained high quality and robust 3D consistency. Existing 3D controllable generative models are designed to synthesize 3D-aware images through a single modality, such as 2D segmentation or sketches, but lack the ability to finely control generated content, such as texture and age. In pursuit of enhancing user-guided controllability, we propose Multi3D, a 3D-aware controllable image synthesis model that supports multi-modal input. Our model can govern the geometry of the generated image using a 2D label map, such as a segmentation or sketch map, while concurrently regulating the appearance of the generated image through a textual description. To demonstrate the effectiveness of our method, we have conducted experiments on multiple datasets, including CelebAMask-HQ, AFHQ-cat, and shapenet-car. Qualitative and quantitative evaluations show that our method outperforms existing state-of-the-art methods. Abstract Image

查看原文本刊更多论文

Multi3D：三维感知多模态图像合成

摘要三维感知图像合成已经实现了高质量和稳健的三维一致性。现有的三维可控生成模型旨在通过二维分割或草图等单一方式合成三维感知图像，但缺乏精细控制生成内容（如纹理和年龄）的能力。为了提高用户引导的可控性，我们提出了支持多模态输入的 3D 感知可控图像合成模型 Multi3D。我们的模型可以使用二维标签图（如分割图或草图）来控制生成图像的几何形状，同时通过文字描述来控制生成图像的外观。为了证明我们方法的有效性，我们在多个数据集上进行了实验，包括 CelebAMask-HQ、AFHQ-cat 和 shapenet-car。定性和定量评估结果表明，我们的方法优于现有的最先进方法。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Computational Visual Media Computer Science-Computer Graphics and Computer-Aided Design

CiteScore

16.90

自引率

5.80%

发文量

243

审稿时长

6 weeks

期刊介绍： Computational Visual Media is a peer-reviewed open access journal. It publishes original high-quality research papers and significant review articles on novel ideas, methods, and systems relevant to visual media. Computational Visual Media publishes articles that focus on, but are not limited to, the following areas: • Editing and composition of visual media • Geometric computing for images and video • Geometry modeling and processing • Machine learning for visual media • Physically based animation • Realistic rendering • Recognition and understanding of visual media • Visual computing for robotics • Visualization and visual analytics Other interdisciplinary research into visual media that combines aspects of computer graphics, computer vision, image and video processing, geometric computing, and machine learning is also within the journal''s scope. This is an open access journal, published quarterly by Tsinghua University Press and Springer. The open access fees (article-processing charges) are fully sponsored by Tsinghua University, China. Authors can publish in the journal without any additional charges.