KVCB: Key-Value cache merging towards consistent behavior for efficient inference of large language models

Abstract

Although key-value (KV) cache eliminates redundant computations in autoregressive generation of large language models (LLMs), its memory cost scales linearly with sequence length, rendering long-context deployment prohibitively expensive. Existing KV cache compression methods primarily prune or merge low-importance KV pairs with long-tail information, yet aggressive compression inevitably excises critical knowledge embedded in the long tail. This degrades generation fidelity and task generalizability. We observe that neighboring KV pairs frequently exhibit highly similar attention patterns during inference, suggesting inherent redundancy. Building upon this insight, we propose KV cache merging towards consistent behavior (KVCB), a novel KV cache compression method for efficient LLM inference. KVCB identifies neighboring KV pairs with consistent behavior through historical attention scores and merges them with weights derived from cumulative historical attention scores. When high-consistency pairs are scarce, KVCB falls back to importance-weighted merging based on temporal position and historical attention scores. To mitigate semantic drift from excessive compression, KVCB incorporates a periodic recovery mechanism that periodically flushes accumulated artifacts for sustained long-context stability. Extensive experiments with Llama-2-7B, Llama-2-7B-Chat, Llama-2-13B-Chat, and CodeLlama-7B on language modeling, long-document question answering, multiple-choice reasoning, and code generation—complemented by ablation studies across model architectures and scales—demonstrate that KVCB consistently matches or outperforms state-of-the-art eviction- and merging-based baselines under fixed memory budgets, effectively balancing compression efficiency with generative fidelity across diverse domains and context lengths.