OASIS: Outlier-Aware KV Cache Clustering for Scaling LLM Inference in CXL Memory Systems

IF 1.4 3区计算机科学 Q4 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

IEEE Computer Architecture Letters Pub Date : 2025-03-07 DOI:10.1109/LCA.2025.3567844

Minseok Seo;Jungi Hyun;Seongho Jeong;Xuan Truong Nguyen;Hyuk-Jae Lee;Hyokeun Lee

引用次数: 0

Abstract

The key-value (KV) cache in large language models (LLMs) now necessitates a substantial amount of memory capacity as its size proportionally grows with the context’s size. Recently, Compute-Express Link (CXL) memory becomes a promising method to secure memory capacity. However, CXL memory in a GPU-based LLM inference platform entails performance and scalability challenges due to the limited bandwidth of CXL memory. This paper proposes OASIS, an outlier-aware KV cache clustering for scaling LLM inference in CXL memory systems. Our method is based on the observation that clustering is effective in trading off between performance and accuracy compared to previous quantization- or selection-based approaches if clustering is aware of outliers. Our evaluation shows OASIS yields 3.6× speedup compared to the case without clustering while preserving accuracy with just 5% of full KV cache.

查看原文本刊更多论文

OASIS：用于扩展CXL存储系统中LLM推理的离群值感知KV缓存聚类

大型语言模型（llm）中的键值（KV）缓存现在需要大量的内存容量，因为它的大小随着上下文的大小成比例地增长。最近，计算机快速链接（CXL）存储器成为一种很有前途的存储容量保护方法。然而，由于CXL内存的带宽有限，基于gpu的LLM推理平台中的CXL内存带来了性能和可伸缩性方面的挑战。本文提出了OASIS，一种用于扩展CXL存储系统中LLM推理的离群值感知KV缓存聚类。我们的方法是基于这样的观察：如果聚类意识到异常值，那么与之前基于量化或选择的方法相比，聚类在性能和准确性之间的权衡是有效的。我们的评估表明，与没有聚类的情况相比，OASIS的速度提高了3.6倍，同时仅在5%的全KV缓存下保持精度。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Computer Architecture Letters COMPUTER SCIENCE, HARDWARE & ARCHITECTURE-

CiteScore

4.60

自引率

4.30%

发文量

期刊介绍： IEEE Computer Architecture Letters is a rigorously peer-reviewed forum for publishing early, high-impact results in the areas of uni- and multiprocessor computer systems, computer architecture, microarchitecture, workload characterization, performance evaluation and simulation techniques, and power-aware computing. Submissions are welcomed on any topic in computer architecture, especially but not limited to: microprocessor and multiprocessor systems, microarchitecture and ILP processors, workload characterization, performance evaluation and simulation techniques, compiler-hardware and operating system-hardware interactions, interconnect architectures, memory and cache systems, power and thermal issues at the architecture level, I/O architectures and techniques, independent validation of previously published results, analysis of unsuccessful techniques, domain-specific processor architectures (e.g., embedded, graphics, network, etc.), real-time and high-availability architectures, reconfigurable systems.