Towards Segmentation-Based Address Translation for LLM Inference

Abstract

As large language models (LLMs) continue to scale in both model size and context length, the efficiency of address translation has become a critical performance bottleneck during inference. Although LLM inference accesses model weights in a layer-wise pattern with substantial reuse across token generations, the expanding KV-cache footprint increasingly contends for TLB capacity, leaving fewer entries to cache weight translations. In this letter, we revisit segmentation-based address translation for model weights to bypass conventional TLB-based translation. Since model weights are read-only and not reclaimed for the lifetime of a serving instance, the segmentation-based address translation is well-suited to this access pattern. We place model weights in a virtually contiguous region backed by a physically contiguous memory, enabling segment-based translation that minimizes address translation overhead. Our evaluation shows that the proposed design improves overall IPC by 2.51 × over the paging baseline by achieving near-zero translation overhead for model weights.