Sync-4D: Monocular 4D reconstruction and generation with Synchronized Canonical Distillation

Abstract

The development of video diffusion models and score distillation techniques has advanced dynamic 3D content generation. However, motion priors from video diffusion models have limited quality and temporal extent. Inspired by motion capture, we propose a text-to-4D framework that generates 4D content using skeletal animations extracted from monocular video. To enhance the 2D diffusion model for temporal-consistent 4D generation, we establish inter-frame token correspondences through canonical coordinate matching and fuse diffusion features. We further propose Synchronized Canonical Distillation (SCD) from a gradient-based perspective. In the score-matching process, SCD computes gradients over articulated models and denoises both the canonical model and motion field synchronously. By accumulating inter-frame and inter-view gradients, SCD mitigates multi-face artifacts and temporal inconsistencies, while diffusion priors further enhance consistency in unobserved regions. Experiments demonstrate that our method outperforms state-of-the-art monocular non-rigid reconstruction and 4D generation methods, achieving a 42.5% lower average Chamfer Distance.