Activity recognition in unconstrained RGB-D video using 3D trajectories

Abstract

Human activity recognition in unconstrained RGB--D videos has extensive applications in surveillance, multimedia data analytics, human-computer interaction, etc, but remains a challenging problem due to the background clutter, camera motion, viewpoint changes, etc. We develop a novel RGB--D activity recognition approach that leverages the dense trajectory feature in RGB videos. By mapping the 2D positions of the dense trajectories from RGB video to the corresponding positions in the depth video, we can recover the 3D trajectory of the tracked interest points, which captures important motion information along the depth direction. To characterize the 3D trajectories, we apply motion boundary histogram (MBH) to depth direction and propose 3D trajectory shape descriptors. Our proposed 3D trajectory feature is a good complementary to dense trajectory feature extracted from RGB video only. The performance evaluation on a challenging unconstrained RGB--D activity recognition dataset, i.e., Hollywood 3D, shows that our proposed method outperforms the baseline methods (STIP-based) significantly, and achieves the state-of-the-art performance.