Spatiotemporal Action Detection Using 2D CNN and 3D CNN

In order to address the low accuracy issue in human spatiotemporal action detection tasks, this study proposes a more effective CNN framework. Like YOWO model, we also use CNN for feature extraction, however, we only utilize the extracted spatiotemporal features for action recognition and the fused features of spatiotemporal and spatial information for action localization. Additionally, in the action localization branch, we make improvements to the original channel fusion and attention mechanism (CFAM). We introduce a combination of convolution and attention mechanisms to selectively replace the traditional convolutions, enabling more effective utilization of the fused features. Finally, in order to make the model more accurate for bounding box regression, we use CIoU loss instead of the offset loss. Results show that our proposed method achieves frame-mAP scores (@IoU 0.5) of 75.73 % and 83.13 % on JHMDB-21 and UCF101–24 datasets, respectively. For video-mAP, we obtain 88.96 %, 85.81 % and 68.59 % at IoU threshold of 0.2,0.5 and 0.75 on JHMDB-21 dataset and 75.05 %, 69.72 % and 48.95 % at IoU threshold of 0.1,0.2 and 0.5 on UCF101–24 dataset.