Enabling Early Gesture Recognition by Motion Augmentation

In real-time gesture recognition algorithms, accurately classifying gestures early, when they are only partially observed, can be advantageous as it minimizes latency and improves user experience. This work investigates a novel approach for improving the results of an early gesture classification model. The method involves augmenting the input sequence of human poses of a partially observed gesture with a series of poses predicted by an auxiliary recurrent neural network sequence-to-sequence motion prediction model before being fed into a random forest gesture classifier. By concatenating the partially observed ground truth sequence with the forecasted motion sequence, we are able to significantly improve early gesture recognition accuracy. When forecasting 25 future frames of a partially observed input gesture sequence of 50 frames, recognition accuracy improves from 45% to 87% on average when evaluated on the MSRC-12 gesture dataset.