Efficient Speaker Naming via Deep Audio-Face Fusion and End-to-End Attention Model

Abstract

Speaker naming has recently received wide attention in identifying the speaking character in a movie video, and it is an extremely challenging topic mainly attributed to the significant variation of facial appearance. Motivated by multimodal applications, we present an efficient speaker naming approach via deep audio-face fusion and end-to-end attention model. First, we start with LSTM-encoding of acoustic feature and VGG-encoding of face images, and then exploit an end-to-end common attention vector by convolution-softmax encoding of their locally concatenated features, whereby the face attention vector can be well discriminated. Further, we apply the low-rank bilinear model to efficiently fuse the face attention vector and acoustic feature vector, whereby the joint audio-face representation can be discriminatively obtained for speaker naming. In addition, we address another acoustic feature representation scheme by convolution-encoding, which can replace LSTM in networks to speed up the training process. The experimental results have shown that our proposed speaker naming approach yields comparative and even better results than the state-of-the-art counterparts.