Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punctuation Restoration

Punctuation restoration is a post-processing task of automatic speech recognition to generate the punctuation marks on un-punctuated transcripts. This paper proposes a deep recurrent neural network architecture with layer-wise multi-head attentions towards better modelling of the contexts from a variety of perspectives in putting punctuations by human writers. The experimental results show that our proposed model significantly outperforms previous state-of-the-art methods in punctuation restoration performances on IWSLT dataset.