Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Abstract

Although deep neural networks (DNNs) have achieved great performance gain, the immense computational cost of DNN model training has become a major block to utilize massive speech data for DNN training. Previous research on DNN training acceleration mostly focussed on hardware-based parallelization. In this paper, node pruning and arc restructuring are proposed to explore model redundancy after a novel lightly discriminative pretraining process. With some measures of node/arc importance, model redundancies are automatically removed to form a much more compact DNN. This significantly accelerates the subsequent back-propagation (BP) training process. Model redundancy reduction can be combined with multiple GPU parallelization to achieve further acceleration. Experiments showed that the combined acceleration framework can achieve about 85% model size reduction and over 4.2 times speed-up factor for BP training on 2 GPUs, at no loss of recognition accuracy.