Make Large Language Models Efficient: A Review

Abstract

Large Language Models (LLMs) have achieved remarkable success across a variety of natural language processing tasks, with larger architectures often exhibiting superior performance. This scaling behavior has fueled intense competition in generative AI, supported by projected investments that exceed ${\$} $ 1 trillion to develop increasingly sophisticated LLMs. This competition has in turn nurtured a vibrant ecosystem, inspiring new open-source models such as DeepSeek, and motivating application developers to harness state-of-the-art LLMs for real-world deployments. However, the extensive memory and computational requirements of large models present serious obstacles for small-medium organizations, leading to significant scalability concerns. This paper offers a comprehensive review of recent techniques to improve LLM efficiency through four categories: parameter-centric, architecture-centric, training-centric and data-centric. For a better understanding of the newcomer’s perspective, it covers the entire lifecycle when developing and deploying LLMs. Thus, this paper is organized around five core tasks: model compression for local deployment, accelerated pre-training to reduce time-to-train, efficient fine-tuning on custom data, optimized inference under resource constraints, and streamlined data preparation. Rather than focusing on broad strategies, we emphasize specialized techniques tailored to each stage of development. By applying targeted optimizations at each phase, the computational overhead can be reduced by 50–95% without compromising the quality of the model, making LLMs more accessible to researchers and practitioners with limited computational resources.