Transfer Learning for Punctuation Prediction

The output from most of the Automatic Speech Recognition system is a continuous sequence of words without proper punctuation. This decreases human readability and the performance of downstream natural language processing tasks on ASR text. We treat the punctuation prediction task as a sequence tagging task and propose an architecture that uses pre-trained BERT embeddings. Our model significantly improves the state of art on the IWSLT dataset. We achieve an overall F1 of 81.4% on the joint prediction of period, comma and question mark.