Deep Randomized Time Warping for Action Recognition

This paper proposes an enhanced Randomized Time Warping (RTW) using CNN features, termed Deep RTW, for motion recognition. RTW is a general extension of Dynamic Time Warping (DTW), widely used for matching and comparing sequential patterns. The basic idea of RTW is to simultaneously calculate the similarities between many pairs of various warped patterns, i.e. Time elastic (TE) features generated by randomly sampling the sequential pattern while retaining their temporal order. This mechanism enables RTW to treat the changes in motion speed flexibly. However, naive TE feature vectors generated from raw images are not expected to have high discriminative power. Besides, the dimension of TE features can increase depending on the number of concatenated images. To address the limitations, we incorporate CNN features extracted from 2D/3D CNNs into the framework of RTW as input to address this issue. Our framework is very simple but effective and applicable to various types of CNN architecture. Extensive experiment on public motion datasets, Jester and Something-Something V2, supports the advantage of our method over the original CNNs.