Action Recognition for Videos by Long-Term Point Trajectory Analysis with Background Removal

Recently, dense trajectories were shown to be an efficient video motion representation for action recognition and achieved state-of-the-art results on a variety of video datasets. This paper improves their performance by taking into account camera motion. To estimate camera motion, the authors use long-term point trajectory analysis to cluster image points and propose an algorithm to find possible background cluster from these clusters according to background nature in a video. Considering the original clusters could not segment the foreground and background very well. The authors optimize the background cluster, and use the cluster to rectify the trajectory. Experimental results on three challenging action datasets (i.e., Hollywood2, Olympic Sports and UCF50) show that the rectified trajectories significantly outperform original dense trajectories.