Optimal transport-based fusion of two-stream convolutional networks for action recognition

Understanding human actions in a given video requires spatial and temporal cues for human action recognition. Several deep learning approaches have been explored to extract effective spatio-temporal features. Specifically, two-stream networks have shown prominent performance due to the efficient capturing of motion information by optical flow estimation methods. Here, spatial and temporal paths with RGB & optical flow inputs, respectively, are trained independently and fused at the softmax layer for the classification of actions. However, the conventional two-stream networks exhibit sub-optimal performance mainly due to two reasons: (i) lack of interaction among the streams and (ii) disregard of diverse distributions of RGB & optical flow while fusion. To overcome these limitations, we propose an optimal transport-based fusion of the two-stream networks for action recognition in order to facilitate the alignment of distributions of two streams. First, feature maps from the last layers of CNN are extracted to preserve the pixel-level correspondence between the streams. Next, we calculate the optimal transportation matrix between the feature maps of spatial and temporal streams to map the features from one distribution to the other. Finally, the transformed features are fused to classify the actions. The effectiveness of the proposed approach is demonstrated on widely used action recognition datasets, namely, UCF-101, HMDB-51, SSV2,and Kinetics-400.