Robust Real-Time Human Action Detection through the Fusion of 3D and 2D CNN

Recent approaches for human action detection often rely on appearance and optical flow networks for frame-level detections before linking them to form action tubes. However, they achieve unsatisfactory performance in real-time due to their huge computational complexity and large parameter usage during training. In this paper, we design and implement a unified end-to-end convolutional neural network (CNN) architecture that consists of two branches, extracting both spatial and temporal information concurrently before predicting bounding boxes and action probabilities from video clips. We also design a novel mechanism that exploits the inter-channel dependencies for an effective fusion of features from the branches. Specifically, we propose a Channel Fusion and Relation-Global Attention (CFRGA) module to aggregate the two features smoothly and model their inter-channel dependencies by considering their global scope structural relation information when inferring attention. We conduct experiments on the untrimmed video dataset, UCF101-24, and achieved impressive results in frame-mAP and video-mAP. The experimental results show that our channel fusion and relation-global attention module contributes to its good performance.