Exploiting Pose Mask Features For Video Action Recognition

Abstract

Existing state-of-the-art approaches in video action recognition mostly adopt RGB images and optical flows as input and neglect the abundant information provided by human poses. In this paper, we present a novel method that highlights the essential area of human body and takes it as complementary input for existing action recognition pipelines. The method, called Pose Mask Network (PMN), leverages a 2D pose estimator to extract heatmaps from frames and utilizes them as Pose Masks on the original images. The Pose Masks are robust to the variance of background and focus on key information of human body. Experiments show that our Pose Mask yields a result far exceeding that of using simple pose representations. More importantly, PMN acts as a supplement to other RGB-based approaches. Combining our PMN with Temporal Segment Network, we obtain state-of-the-art performance on the HMDB51 and JHMDB datasets.