Improving Unsupervised Acoustic Word Embeddings using Speaker and Gender Information

For many languages, there is little or no labelled speech data available for training speech processing models. In zero-resource settings where unlabelled speech audio is the only available resource, speech applications for search, discovery and indexing often need to compare speech segments of different durations. Acoustic word embeddings are fixed dimensional representations of variable length speech sequences, allowing for efficient comparisons. Unsupervised acoustic word embedding models often still retain nuisance factors such as a speaker's identity and gender. Here we investigate how to improve the invariance of unsupervised acoustic embeddings to speaker and gender characteristics. We assume that speaker and gender labels are available for the untranscribed training data. We then consider two different methods for normalising out these factors: speaker and gender conditioning, and adversarial training. We apply both methods to two unsupervised embedding models: a recurrent neural network (RNN) autoencoder and a RNN correspondence autoencoder. In a word discrimination task, we find little benefit by explicitly normalising the embeddings to speaker and gender on English data. But on Xitsonga, substantial improvements are achieved. We speculate that this is due to the higher number of speakers present in the unlabelled Xitsonga training data.