Early Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition has attracted much attention in computer vision. Recently, Graph Convolutional Networks (GCNs) with multi-stream fusion strategies have obtained remarkable performance. Most of these models make decisions of action recognition by merging the prediction scores of multiple streams, while ignoring the complementary properties of different data streams for building representative features. In this paper, we propose a novel Early Fusion Graph Convolutional Network (EF-GCN), which fuses hidden features extracted from multiple skeleton data streams at different levels to enhance the discriminative power of features. Unlike the previous GCN-based models that train networks corresponding to different streams independently, all the subnetworks in the proposed EF-GCN are jointly learned in an end-to-end manner. Experiments conducted on two skeleton datasets (i.e., NTU-RGB+D and NTU-120 RGB+D) show the superior performance of EF-GCN.