Occlusion-aware multi-person pose estimation with keypoint grouping and dual-prompt guidance in crowded scenes

Abstract

Multi-person pose estimation (MPPE) in crowded scenes is a challenging task due to severe keypoint occlusions. Although great progress has been made in learning effective joint features for MPPE, existing methods still have two problems. (1) They seldom consider the movement characteristics of human joints and fail to adopt distinct processing strategies to describe different types of joints. (2) They only use simple joint names as text prompts, failing to mine other informative text hints to represent detailed joint situations. To address these two problems, in this paper we propose an occlusion-aware MPPE method by exploring keypoint grouping and dual-prompt guidance (KDG). KDG adopts a distillation learning framework which contains a student network and a teacher network. In the student network, we perform instance decoupling and propose a keypoint grouping strategy to learn global and local context features for two types of joints by considering their movement flexibility. In the teacher network, we introduce the vision-language model to represent the detailed joint situations and explore dual prompts, i.e., rough body part prompts and fine-grained joint prompts, to align text and visual features. Finally, we design loss functions to train the whole network and effectively transfer the rich vision-language knowledge contained in the teacher network to the student network. Experimental results on two benchmark datasets demonstrate the superiority of our KDG over state-of-the-art methods for MMPE in crowded and occluded scenes. The source codes are available at https://github.com/stc-cqupt/KDG.