Making Locality-Aware GEMM Compatible With Page-Granularity Placement on Chiplet GPUs

Abstract

Multi-chiplet GPUs serve each memory access either from the chiplet’s nearby HBM stack or from a remote stack across the silicon interposer, and locality-aware data placement is a common way to keep accesses local. We show that this placement is often not achievable on current GPUs, since data is placed at page granularity (4 KB hardware interleaving or 2 MB GPU pages), while the locality-optimal per-chiplet slices are finer than, or not multiples of, the page granularity. For the 67% of representative LLM GEMMs in this regime, adopting the placement on current systems leaves 75% of HBM accesses remote on a four-chiplet GPU. To address this, we propose Chiplet-Contiguous Layout (CCL), a global memory layout that contiguously stores the data consumed by each chiplet, so that the locality-aware placement becomes realizable with 4 KB placement granularity, without changes to the operating system or hardware. Across representative GEMMs from Qwen 3 30B and Llama 3.1 70B, CCL reduces remote HBM traffic by 9.5 × over 4 KB interleaving and by 2.2 × over coarse locality-aware placement, improving energy efficiency by 17% and 9%, respectively.