LLM KV Cache Storage Using CXL Memory

Abstract

As the memory need for KV cache grows rapidly with the advent of multi-turn inferences and extremely long context requirements, limited GPU memory has become a critical bottleneck. While offloading the KV cache to CPU DRAM or SSDs has been explored, these solutions face limitations in capacity, latency, and scalability. Compute Express Link (CXL) memory offers a promising alternative that provides low-latency, disaggregation, and scalability benefits. In this paper, we evaluate KV cache offloading using various novel CXL memory architectures to compare with existing solutions. We tested memory expanders, CMM-D and CMM-H, and memory pooling solutions, including CMM-B and CMM-MP. For performance evaluation, vLLM, coupled with the LMCache framework, is employed to characterize five offloading targets. This comprehensive study provides insights into KV cache offloading tiers that can be used by end users to optimize their AI data center system architectures. CXL memory devices show significant performance benefits for KV cache offloading, with CMM-MP’s novel memory pooling architecture offering a particularly scalable and cost-efficient solution.