Transformer-based weakly supervised 3D human pose estimation

Abstract

Deep learning-based 3D human pose estimation methods typically require large amounts of 3D pose annotations. However, due to limitations in data quality and the scarcity of 3D labeled data, researchers have adopted weak supervision methods to reduce the demand for annotated data. Compared to traditional approaches, Transformers have recently achieved remarkable success in 3D human pose estimation. Leveraging their powerful modeling and generalization capabilities, Transformers effectively capture patterns and features in the data, even under limited data conditions, mitigating the issue of data scarcity. Nonetheless, the Transformer architecture struggles to capture long-term dependencies and spatio-temporal correlations between joints when processing spatio-temporal features, which limits its ability to model temporal and spatial relationships comprehensively. To address these challenges and better utilize limited labeled data under weak supervision, we proposed an improved Transformer-based model. By grouping joints according to body parts, we enhanced the spatio-temporal correlations between joints. Additionally, the integration of LSTM captures long-term dependencies, improving temporal sequence modeling and enabling the generation of accurate 3D poses from limited data. These structural improvements, combined with weak supervision strategies, enhance the model’s performance while reducing the reliance on extensive 3D annotations. Furthermore, a multi-hypothesis strategy and temporal smoothness consistency constraints were employed to regulate variations between adjacent time steps. Comparisons on the Human3.6M and HumanEva datasets validate the effectiveness of our approach.