Gesture Class Prediction by Recurrent Neural Network and Attention Mechanism

Our objective is to develop a machine-learning model that allows a virtual agent to automatically perform appropriate communicative gestures. Our first step is to compute when a gesture should be performed. We express this as classification problem. We initially split the data into NoGesture class and HasGesture class. We develop a model based on recurrent neural network with attention mechanism to compute the class based on the speech prosody. We apply the model on a dialog corpus segmented into different gesture classes and gesture phases. We treat the prosody as the input sequence and the gesture classes as the output sequence.