Challenges and Opportunities of Language Representation Model

Abstract

Pre-trained distribution natural language representation has been shown to be effective for improving many natural language processing tasks. However, recent research shows that the pre-trained language model has obvious defects in robustness, interpretability and so on. This survey reviews the development of natural language representation model in detail, including but not limited to Word2Vec, Embeddings from Language Models, and Bidirectional Encoder Representations from Transformers. Furthermore, this paper analyzes advantage and disadvantages of existing models from multiple perspectives. Finally, the potential challenges of natural language representation model have been discussed in this survey.