Reducing LLM Inference Memory Bandwidth via Frequent Exponent Value Encoding

IF 2 3区计算机科学 Q4 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

IEEE Computer Architecture Letters Pub Date : 2026-01-01 Epub Date: 2026-03-06 DOI:10.1109/LCA.2026.3671166

Maxwell Michalec;Swamit Tannu;Gurindar Sohi

引用次数: 0

Abstract

We consider the issue of the memory bandwidth required for the transfer of weights of an LLM between the memory and the processor (CPU or GPU). Observing that a few exponent values dominate for FP8 and BF16 floating-point formats, we propose an alternate (compressed) representation and storage of the exponents in memory, and a design for reconstructing the full, exact weight values in the processor. The proposed approach reduces the memory bandwidth needed by up to 10% and 30% for LLMs using weight in FP8 and BF16 formats, respectively, without changing any of the computation carried out by the original LLM inference software.

查看原文本刊更多论文

通过频繁指数值编码减少LLM推理存储器带宽

我们考虑了在内存和处理器（CPU或GPU）之间传递LLM权重所需的内存带宽问题。观察到FP8和BF16浮点格式中一些指数值占主导地位，我们提出了一种替代的（压缩的）表示和内存中指数的存储，以及在处理器中重建完整的、精确的权重值的设计。所提出的方法在不改变原始LLM推理软件进行的任何计算的情况下，分别将FP8和BF16格式的LLM使用权重所需的内存带宽减少了10%和30%。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Computer Architecture Letters COMPUTER SCIENCE, HARDWARE & ARCHITECTURE-

CiteScore

4.60

自引率

4.30%

发文量

期刊介绍： IEEE Computer Architecture Letters is a rigorously peer-reviewed forum for publishing early, high-impact results in the areas of uni- and multiprocessor computer systems, computer architecture, microarchitecture, workload characterization, performance evaluation and simulation techniques, and power-aware computing. Submissions are welcomed on any topic in computer architecture, especially but not limited to: microprocessor and multiprocessor systems, microarchitecture and ILP processors, workload characterization, performance evaluation and simulation techniques, compiler-hardware and operating system-hardware interactions, interconnect architectures, memory and cache systems, power and thermal issues at the architecture level, I/O architectures and techniques, independent validation of previously published results, analysis of unsuccessful techniques, domain-specific processor architectures (e.g., embedded, graphics, network, etc.), real-time and high-availability architectures, reconfigurable systems.