BlockPIM: Enabling Sparse LLM Inference on Dense PIM Architectures

Abstract

Large Language Models (LLMs) have intensified the memory wall during inference, where performance is increasingly limited by memory bandwidth rather than computation. While Processing-In-Memory (PIM) and model compression techniques mitigate this issue, existing sparse PIM designs rely on CSR/COO formats that assume ultra-high sparsity (e.g., $> 90\%$), which is difficult to achieve in practical LLMs without significant accuracy loss. Moreover, fine-grained indexing introduces substantial metadata overhead, diminishing the benefits of pruning, especially under low-bit quantization. To address this mismatch, we propose BlockPIM, a PIM architecture that employs hardware-aligned coarse-grained pruning to efficiently execute sparse LLMs on standard dense computation units. BlockPIM replaces per-element indices with compact bitmaps and contiguous non-zero blocks, enabling sparse workloads to be processed directly on dense execution pipelines without complex index decoding or irregular control flow. Experimental results show that BlockPIM achieves up to 8.0× and 3.6× speedups over the NPU and state-of-the-art sparse PIM baselines, respectively, while reducing energy consumption by up to 95.2% and providing superior memory efficiency under practical accuracy constraints.