Single image and video generation using a receptive diffusion model with convolutional spatiotemporal blocks

Abstract

The generation of images from a single natural image/video has garnered significant attention due to its broad applications. However, existing methods training on a single input image or video face two key limitations. First, GAN-based approaches, relying on multiple models trained at progressively increasing scales, often lead to error accumulation and artifacts in generated results. Second, while diffusion models offer superior quality and diversity, they require extensive training time for a single input and are limited to generation tasks without the ability to edit existing images or videos. To address these challenges, we propose a Unified Diffusion Model for Single Image/Video Training, named Union, achieving a balanced trade-off between computational efficiency and visual quality. Specifically, we introduce: (1) a unified model trained at a single scale, avoiding the error accumulation seen in multi-scale models; and (2) a novel Receptive DDPM framework with convolutional spatiotemporal blocks (CS-Block) that learns patch distribution of a natural image rather than simple image replication. The CS-Block uses ConvNext and spatiotemporal attention mechanisms to capture local and global relationships in temporal and frequency domains, enabling efficient adaptation to the patch-level receptive field of natural images and videos. Extensive experiments across image and video tasks demonstrate that Union outperforms other methods, achieving the best LPIPS score on the public Places50 dataset and excelling in high-resolution video generation, providing an optimal balance between computational cost and performance. The training and generated images/videos are available at: https://github.com/hylneu/union.git.