Temporal diffuser: Timing scale-aware modulation for sign language production

Abstract

Recent advances in Sign Language Production (SLP) highlight denoising diffusion models as promising alternatives to traditional autoregressive methods. Most existing approaches follow a two-stage pipeline that encodes sign motion into discrete latent codes, often sacrificing Space–Time fidelity and requiring gloss annotations or complex codebooks. Transformer-based models aim to simplify this, but often produce overly smooth, unnatural motions. We introduce Sign Language Production with Scale-Aware Modulation (SignSAM), a novel single-stage, gloss-free SLP framework that directly synthesizes motion in continuous space, preserving fine temporal details. At its core is a Space–Time U-Net that learns compact temporal features by jointly downscaling the frame and sign feature dimensions, thereby reducing computational cost compared to a no-pyramid UNet or a pyramid UNet without consistency between dimensions. To further enhance temporal precision, we propose a Timing Scale-Aware Modulation module that fuses multiscale temporal resolutions for better motion coherence. Experiments on PHOENIX14T and How2Sign show that SignSAM achieves state-of-the-art (SOTA) fluency, accuracy, and naturalness, offering an efficient and expressive solution for SLP. Our project homepage is https://kha-kim-thuy.github.io/SLP-Demo/.