Fine-Grained Egocentric Action Recognition with Multi-Modal Unsupervised Domain Adaptation

Fine-grained egocentric action recognition has made significant progress because of the advancement of supervised learning. Some real-world applications require the network trained on one dataset to perform well on another unlabeled dataset due to the difficulty of annotating new data. However, due to the disparity in dataset distributions, i.e. domain shift, the network is unable to retain its good performance across datasets. Therefore, in this paper, we use unsupervised domain adaptation to address this difficult challenge, i.e. training a model on labeled source data such that it can be directly used on unlabeled target data with the same categories. First, we use Transformer to capture spatial information, and then we propose a temporal attention module to model temporal interdependence. In consideration of the fact that multi-modal data provides more kinds of important information, we build a tri-stream network for spatio-temporal information fusion. Finally, we align source data with target data using adversarial learning. Our network outperforms the baselines on the largest egocentric dataset, the EPIC-KITCHENS-100 dataset.