High-Bandwidth Flash for KV Caches: Endurance and Performance Implications

Abstract

During Large Language Model (LLM) inference, the key-value (KV) cache is a critical bottleneck in both capacity and memory bandwidth as context lengths surge. While High-Bandwidth Memory (HBM) can meet current bandwidth demands, its capacity has not kept pace with the rapidly growing footprint of the KV cache. Consequently, KV caches are frequently offloaded to CPU memory for higher capacity, yet performance remains constrained by limited interconnect bandwidth. High-Bandwidth Flash (HBF) has emerged as a promising alternative, offering sequential read bandwidth comparable to HBM4 with up to an order-of-magnitude higher capacity. Although Flash memory has limited write endurance, KV caches exhibit a favorable access pattern, being written once and read many times during inference. Our analysis demonstrates that the KV cache read-to-write ratio increases with longer context lengths, particularly with larger output sequences. This characteristic enables KV caches to be stored in HBF while maintaining an acceptable device lifetime, addressing both the capacity and bandwidth challenges of contemporary LLM systems.