Distributed training of large language models: A survey

The emergence of large language models (LLMs) such as ChatGPT has opened up groundbreaking possibilities, enabling a wide range of applications in diverse fields, including healthcare, law, and education. A recent research report highlighted that the performance of these models is often closely tied to their parameter scale, raising a pressing question: how can we effectively train LLMs? This concern is at the forefront of many researchers’ minds. Currently, several distributed training frameworks, such as Megatron-LM and DeepSpeed, are widely used. In this paper, we provide a comprehensive overview of the current state of LLMs, beginning with an introduction to their development status. We then dig into the common parallel strategies employed in LLM distributed training, followed by an examination of the underlying technologies and frameworks that support these models. Next, we discuss the state-of-the-art optimization techniques used in LLMs. Finally, we summarize some key challenges and limitations of current LLM training methods and outline potential future directions for the development of LLMs.