DayPQ: Dynamic Layerwise Pruning and Quantization for LLM Inference Acceleration

Abstract

The deployment of the prevailing large language models (LLMs) on resource-constrained hardware faces critical challenges due to their massive computational and memory requirements. Pruning and quantization (PQ), as two predominant techniques for LLM acceleration, face the challenge of fully preserving model capabilities while achieving significant acceleration in practice. Static PQ irreversibly discards potentially critical parameters and degrades precision across diverse inputs, which must be compensated through fine-tuning. Dynamic PQ preserves accuracy but incurs prohibitive hardware overhead and additional latency from runtime adaptations. Achieving low-latency and overhead-efficient dynamic PQ requires the co-design of adaptive compression tightly integrated with hardware. In this work, we present DayPQ, an algorithm-hardware co-optimization technique for efficient LLM inference: self-adaptive norm-based pruning and bias-shifting FP8 quantization at the algorithm level, synergized with a multiplication-free shift–add PE array and switchable dataflow at the hardware level, achieving dynamic acceleration with the negligible accuracy loss and zero additional latency. DayPQ enables adaptive PQ by intrinsically exploiting normalization operations from the model’s inference process instead of extra operations. Simultaneously, it establishes dynamic orthogonality between sparsity and quantization, where these dual compression mechanisms synergistically enhance the redundancy reduction beyond mere additive effects of independent operations. Our evaluation shows that DayPQ obtains an average $3.13\times $ higher energy efficiency and $2.03\times $ performance boost than three state-of-the-art (SOTA) accelerators.