Joint Optimization of Classification and Clustering for Deep Speaker Embedding

Abstract

This paper proposes a method to train deep speaker embed-dings end-to-end that jointly optimizes classification and clustering. A large margin softmax loss is used to reduce classification errors. A novel large margin Gaussian mixture loss is proposed to improve clustering. With the joint optimization, the learned embeddings capture segment-level acoustic representation from variable-length speech segments to discriminate between speakers and to replicate densities of speaker clusters. We compare performance with alternative methods on large-scale text-independent speaker recognition dataset VoxCeleb1 [1] and observe that it outperforms those methods significantly, achieving new state-of-the-art results on the dataset. Moreover, because of the joint optimization, this method exhibits faster and better convergence than using classification loss alone. Our results suggest great potential of joint optimization of classification and clustering for speaker verification and identification.