STAR-SNR: spatial–temporal adaptive regulation and SNR optimization for few-shot video generation

Abstract

In recent years, text-to-image generation technology based on diffusion models has made significant progress, but extending it to the field of video generation, especially under few-shot conditions, still faces huge challenges. Existing methods usually rely on a large amount of text-video pair data or consume a lot of training resources. Based on this, this paper proposes a new few-shot video generation framework, STAN-SNR, which combines spatio-temporal feature regulation, feature scrolling enhancement and dynamic signal-to-noise ratio (SNR) weighting strategies, using 8–16 videos on a single A6000 training, effectively improving the quality and efficiency of video generation and reducing the amount of calculation. Specifically, the spatio-temporal feature regulation module effectively extracts spatio-temporal features and reduces computational complexity. The feature scrolling enhancement module enhances the ability to capture local features to avoid overfitting. In addition, the dynamic SNR weighting strategy adjusts the loss calculation according to the time step, which improves the convergence speed of the model, which is 2.44 times faster compared with the baseline model. The experimental results show that the STAN-SNR framework generates videos with higher text alignment, consistency, and diversity under few-shot conditions.