Semi-supervised training strategies for deep neural networks

Abstract

Use of both manually and automatically labelled data for model training is referred to as semi-supervised training. While semi-supervised acoustic model training has been well-explored in the context of hidden Markov Gaussian mixture models (HMM-GMMs), the re-emergence of deep neural network (DNN) acoustic models has given rise to some novel approaches to semi-supervised DNN training. This paper investigates several different strategies for semi-supervised DNN training, including the so-called ‘shared hidden layer’ approach and the ‘knowledge distillation’ (or student-teacher) approach. Particular attention is paid to the differing behaviour of semi-supervised DNN training methods during the cross-entropy and sequence training phases of model building. Experimental results on our internal study dataset provide evidence that in a low-resource scenario the most effective semi-supervised training strategy is ‘naive CE’ (treating manually transcribed and automatically transcribed data identically during the cross entropy phase of training) followed by use of a shared hidden layer technique during sequence training.