Improved Viseme Recognition using Generative Adversarial Networks

Abstract

The proliferation of convolutional neural networks (CNN) has resulted in increased interest in the field of visual speech recognition (VSR). However, while VSR for word-level and sentence-level classification has received much of this attention, recognition of visemes has remained relatively unexplored. This paper focuses on the visemic approach for VSR as it can be used to build language-independent models. Our method employs generative adversarial networks (GANs) to create synthetic images that are used for data augmentation. VGG16 is used for classification both before and after augmentation. The results obtained prove that data augmentation using GANs is a viable technique for improving the performance of VSR models. Augmenting the dataset with images generated using the Progressive Growing Generative Adversarial Network (PGGAN) model led to an average increase in test accuracy of 3.695% across speakers. An average increase in test accuracy of 2.59% was achieved by augmenting the dataset using images generated by the conditional Deep Convolutional Generative Adversarial Network (DCGAN) model.