Robust Multi-Modal Cues for Dyadic Human Interaction Recognition

Activity analysis methods usually tend to focus on elementary human actions but ignore to analyze complex scenarios. In this paper, we focus particularly on classifying interactions between two persons in a supervised fashion. We propose a robust multi-modal proxemic descriptor based on 3D joint locations, depth and color videos. The proposed descriptor incorporates inter-person and intra-person joint distances calculated from 3D skeleton data and multi-frame dense optical flow features obtained from the application of temporal Convolutional neural networks (CNN) on depth and color images. The descriptors from the three modalities are derived from sparse key-frames surrounding high activity content and fused using a linear SVM classifier. Through experiments on two publicly available RGB-D interaction datasets, we show that our method can efficiently classify complex interactions using only short video snippet, outperforming existing state-of-the-art results.