Domain-Specific Chinese Transformer-XL Language Model with Part-of-Speech Information

Abstract

Language models hope to use more context to predict the next word. However, not all words in the context are related to the next word and are effective for prediction. The language model based on the attention mechanism can select more useful word representations from the context and efficiently use long-term historical information. In this paper, we will apply Transformer-XL language model to Chinese automatic speech recognition in a specific domain. We add part-of-speech information for domain adaptation. First, we construct a Chinese corpus dataset in a specific domain. And by collecting common vocabulary and extracting new words in the domain, we also construct a domain vocabulary. Then, the Chinese word boundary information is added to the Transformer-XL language model to make the model can better adapt to the characteristics of the domain. Finally, our experimental results show that the method is effective on the dataset we provided. It can further reduce the Character Error Rate (CER) in speech recognition.