The evolution of language models: From N-Grams to LLMs, and beyond

Abstract

In the last couple of decades language models and artificial intelligence technologies have had significant improvements. Along with computer vision and image processing models, large language models (LLMs) are expected to have big impacts on how AI technologies will evolve. As such, it is important to study how language models have advanced since their inception; and more importantly how they will grow in the future.

In this article, we provide an overview of the evolution of language models. We start with early statistical and rule-based models. The advancement of language models are discussed all the way to nowadays transformer-based multimodal models (MM-LLMs). We discuss the shortcomings of the current language models and various aspects of the models that need to be improved upon. We also highlight the latest research trends in NLP. Furthermore, we pinpoint important aspects of language models and AI technologies that need further attention. This overview paper provides valuable insights about the progression of language models. It can be motivational and helpful for advancing the state-of-art language models.