Towards realizing gesture-to-speech conversion with a HMM-based bilingual speech synthesis system

Abstract

This paper realizes a gesture-to-speech conversion system to solve the communication problem between healthy people and speech disorders. An improved speeded up robust features (SURF) algorithm is adopted for static gesture recognition by combining Kinect sensor. Meanwhile, a Hidden Markov Model (HMM) based Mandarin-Tibetan bilingual speech synthesis system is developed by using speaker adaptive training. A set of semantic rules is designed for the static gestures. Chinese or Tibetan context-dependent labels of recognized static gestures are generated according to the semantic rules. The recognized gestures are finally converted to the Mandarin or Tibetan by using the Mandarin-Tibetan bilingual speech synthesis system with the context-dependent labels. Tests show that the static gesture recognition rate of the designed system achieves 97.1%. Subjective evaluation demonstrates that synthesized speech can get 4.0 of the mean opinion score (MOS) on synthesized speech.