通过频繁指数值编码减少LLM推理存储器带宽

IF 2 3区计算机科学 Q4 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

IEEE Computer Architecture Letters Pub Date : 2026-01-01 Epub Date: 2026-03-06 DOI:10.1109/LCA.2026.3671166

Maxwell Michalec;Swamit Tannu;Gurindar Sohi

{"title":"通过频繁指数值编码减少LLM推理存储器带宽","authors":"Maxwell Michalec;Swamit Tannu;Gurindar Sohi","doi":"10.1109/LCA.2026.3671166","DOIUrl":null,"url":null,"abstract":"We consider the issue of the memory bandwidth required for the transfer of weights of an LLM between the memory and the processor (CPU or GPU). Observing that a few exponent values dominate for FP8 and BF16 floating-point formats, we propose an alternate (compressed) representation and storage of the exponents in memory, and a design for reconstructing the full, exact weight values in the processor. The proposed approach reduces the memory bandwidth needed by up to 10% and 30% for LLMs using weight in FP8 and BF16 formats, respectively, without changing any of the computation carried out by the original LLM inference software.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"25 1","pages":"85-88"},"PeriodicalIF":2.0000,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Reducing LLM Inference Memory Bandwidth via Frequent Exponent Value Encoding\",\"authors\":\"Maxwell Michalec;Swamit Tannu;Gurindar Sohi\",\"doi\":\"10.1109/LCA.2026.3671166\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We consider the issue of the memory bandwidth required for the transfer of weights of an LLM between the memory and the processor (CPU or GPU). Observing that a few exponent values dominate for FP8 and BF16 floating-point formats, we propose an alternate (compressed) representation and storage of the exponents in memory, and a design for reconstructing the full, exact weight values in the processor. The proposed approach reduces the memory bandwidth needed by up to 10% and 30% for LLMs using weight in FP8 and BF16 formats, respectively, without changing any of the computation carried out by the original LLM inference software.\",\"PeriodicalId\":51248,\"journal\":{\"name\":\"IEEE Computer Architecture Letters\",\"volume\":\"25 1\",\"pages\":\"85-88\"},\"PeriodicalIF\":2.0000,\"publicationDate\":\"2026-01-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Computer Architecture Letters\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/11422880/\",\"RegionNum\":3,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2026/3/6 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"Q4\",\"JCRName\":\"COMPUTER SCIENCE, HARDWARE & ARCHITECTURE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Computer Architecture Letters","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/11422880/","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/3/6 0:00:00","PubModel":"Epub","JCR":"Q4","JCRName":"COMPUTER SCIENCE, HARDWARE & ARCHITECTURE","Score":null,"Total":0}

引用次数: 0

摘要

我们考虑了在内存和处理器（CPU或GPU）之间传递LLM权重所需的内存带宽问题。观察到FP8和BF16浮点格式中一些指数值占主导地位，我们提出了一种替代的（压缩的）表示和内存中指数的存储，以及在处理器中重建完整的、精确的权重值的设计。所提出的方法在不改变原始LLM推理软件进行的任何计算的情况下，分别将FP8和BF16格式的LLM使用权重所需的内存带宽减少了10%和30%。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Reducing LLM Inference Memory Bandwidth via Frequent Exponent Value Encoding

We consider the issue of the memory bandwidth required for the transfer of weights of an LLM between the memory and the processor (CPU or GPU). Observing that a few exponent values dominate for FP8 and BF16 floating-point formats, we propose an alternate (compressed) representation and storage of the exponents in memory, and a design for reconstructing the full, exact weight values in the processor. The proposed approach reduces the memory bandwidth needed by up to 10% and 30% for LLMs using weight in FP8 and BF16 formats, respectively, without changing any of the computation carried out by the original LLM inference software.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

IEEE Computer Architecture Letters COMPUTER SCIENCE, HARDWARE & ARCHITECTURE-

CiteScore

4.60

自引率

4.30%

发文量

期刊介绍： IEEE Computer Architecture Letters is a rigorously peer-reviewed forum for publishing early, high-impact results in the areas of uni- and multiprocessor computer systems, computer architecture, microarchitecture, workload characterization, performance evaluation and simulation techniques, and power-aware computing. Submissions are welcomed on any topic in computer architecture, especially but not limited to: microprocessor and multiprocessor systems, microarchitecture and ILP processors, workload characterization, performance evaluation and simulation techniques, compiler-hardware and operating system-hardware interactions, interconnect architectures, memory and cache systems, power and thermal issues at the architecture level, I/O architectures and techniques, independent validation of previously published results, analysis of unsuccessful techniques, domain-specific processor architectures (e.g., embedded, graphics, network, etc.), real-time and high-availability architectures, reconfigurable systems.