Hyunwoo Kim, Itai Lang, Noam Aigerman, Thibault Groueix, Vladimir G. Kim, Rana Hanocka
{"title":"MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation","authors":"Hyunwoo Kim, Itai Lang, Noam Aigerman, Thibault Groueix, Vladimir G. Kim, Rana Hanocka","doi":"arxiv-2408.14899","DOIUrl":null,"url":null,"abstract":"We propose MeshUp, a technique that deforms a 3D mesh towards multiple target\nconcepts, and intuitively controls the region where each concept is expressed.\nConveniently, the concepts can be defined as either text queries, e.g., \"a dog\"\nand \"a turtle,\" or inspirational images, and the local regions can be selected\nas any number of vertices on the mesh. We can effectively control the influence\nof the concepts and mix them together using a novel score distillation\napproach, referred to as the Blended Score Distillation (BSD). BSD operates on\neach attention layer of the denoising U-Net of a diffusion model as it extracts\nand injects the per-objective activations into a unified denoising pipeline\nfrom which the deformation gradients are calculated. To localize the expression\nof these activations, we create a probabilistic Region of Interest (ROI) map on\nthe surface of the mesh, and turn it into 3D-consistent masks that we use to\ncontrol the expression of these activations. We demonstrate the effectiveness\nof BSD empirically and show that it can deform various meshes towards multiple\nobjectives.","PeriodicalId":501174,"journal":{"name":"arXiv - CS - Graphics","volume":"26 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-08-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Graphics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2408.14899","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
We propose MeshUp, a technique that deforms a 3D mesh towards multiple target
concepts, and intuitively controls the region where each concept is expressed.
Conveniently, the concepts can be defined as either text queries, e.g., "a dog"
and "a turtle," or inspirational images, and the local regions can be selected
as any number of vertices on the mesh. We can effectively control the influence
of the concepts and mix them together using a novel score distillation
approach, referred to as the Blended Score Distillation (BSD). BSD operates on
each attention layer of the denoising U-Net of a diffusion model as it extracts
and injects the per-objective activations into a unified denoising pipeline
from which the deformation gradients are calculated. To localize the expression
of these activations, we create a probabilistic Region of Interest (ROI) map on
the surface of the mesh, and turn it into 3D-consistent masks that we use to
control the expression of these activations. We demonstrate the effectiveness
of BSD empirically and show that it can deform various meshes towards multiple
objectives.