A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning

Abstract

High-dimensional data such as natural images or speech signals exhibit some form of regularity, preventing their dimensions from varying independently. This suggests that there exists a smaller dimensional latent representation from which the high-dimensional observed data were generated. Uncovering the hidden explanatory features of complex data is the goal of representation learning, and deep latent variable generative models have emerged as promising unsupervised approaches. In particular, the variational autoencoder (VAE) [1, 2], which is equipped with both a generative and inference model, allows for the analysis, transformation, and generation of various types of data. Over the past few years, the VAE has been extended in many ways, including for dealing with data that are either multimodal [3] or dynamical (i.e., sequential) [4]. In this talk, we will present a multimodal and dynamical VAE (MDVAE) applied to unsupervised audiovisual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities (e.g., the speaker's lip movements) from those that are specific to each modality (e.g., the speaker's pitch variation or eye movements). A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence (e.g., the speaker's identity or global emotional state). The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two steps. In the first step, a vector quantized VAE (VQ-VAE) [5] is learned independently for each modality, without temporal modeling. The second step consists in learning the MDVAE, whose inputs are the intermediate representations of the VQ-VAE before quantization. The disentanglement between static versus dynamical and modality-specific versus shared information occurs during this second training stage. Experimental results will be presented, featuring what characteristics of the audiovisual speech data are encoded within the different latent spaces, how the proposed multimodal model can be beneficial compared with a unimodal one, and how the learned representation can be leveraged to perform downstream tasks.