Text-driven 3D human motion generation for pose estimation using dual-transformer architecture

Abstract

Text-to-motion generation has made significant progress in recent years. However, existing approaches struggle to generate high-quality 3D human motions that effectively capture pose estimation. These limitations are due to weak pose estimation and limited skeletal modeling. To address these limitations, we propose DT3DPE (Dual-Transformer for 3D Pose Estimation), a framework that integrates pose estimation to generate text-aligned, realistic 3D human motions. The proposed approach introduces residual vector quantization with additional layers for encoding pose tokens, enabling the capture of fine-grained details in body dynamics. Furthermore, DT3DPE employs a dual-transformer architecture, consisting of a masked transformer for motion token prediction and a residual transformer for refining motion details. This dual-transformer architecture allows the model to generate high-fidelity 3D human poses with precise body joint positioning and smooth temporal transitions. The experimental results on HumanML3D and KIT-ML datasets demonstrate that DT3DPE outperforms existing state-of-the-art methods in text-driven 3D human motion generation. Our code is available at https://github.com/swerizwan/DT3DPE.