COTA-motion: Controllable image-to-video synthesis with dense semantic trajectories

Abstract

Motion transfer, which aims to animate an object in a static image by transferring motion from a reference video, remains a fundamental yet challenging task in content creation. While recent diffusion-based image-to-video models offer fine-grained control over visual appearance, most existing methods rely on ambiguous text prompts or coarse drag-based motion cues, making it difficult to achieve accurate and consistent motion synthesis. To address these limitations, we propose COTA-Motion, a general framework for controllable image-to-video motion transfer. Our method leverages a dense trajectory-based semantic representation extracted from the driving video to provide explicit motion guidance. Specifically, we segment the salient object and extract its point-wise trajectories across frames. These trajectories are enriched with semantic embeddings and reprojected into a spatial-temporal tensor, forming the motion embedding. To utilize this motion representation, we introduce the COTA Adapter, which integrates image content with semantic trajectories via cross-attention, enabling accurate and flexible control over the generated motion. At inference, we further incorporate an alignment module to address discrepancies between the input image and motion cues, ensuring spatial consistency. Built upon a pre-trained video diffusion model, COTA-Motion only requires lightweight fine-tuning on a small set of videos, and it enables high-quality, controllable motion transfer from video to image. Extensive experiments demonstrate the effectiveness of our approach in generating visually coherent and motion-aligned video outputs.