FlexNM: A Fully Flexible SpMM Accelerator for N:M Structured Sparsity

Abstract

N:M structured pruning is widely adopted in modern transformer-based deep learning workloads to reduce model size and enable efficient Sparse-Dense Matrix Multiplication (SpMM). However, existing accelerators either support limited N:M configurations or require hardware regeneration for different sparsity patterns. This limitation prevents efficient acceleration under optimal sparsity patterns, forcing suboptimal N:M configurations or hardware redesign, and resulting in degraded model accuracy. To address this issue, we propose FlexNM, a fully flexible N:M structured SpMM accelerator. FlexNM employs a unified hardware architecture that supports a diverse range of N:M values, enabling direct acceleration of workloads at their optimal N:M sparsity configurations. To achieve this flexibility, FlexNM employs a novel asymmetric adder-tree (AAT) mapping scheme that efficiently supports diverse values of N. It also incorporates a bitmap-based compression scheme to reduce memory overhead and enhance N:M flexibility. FPGA experimental results show that FlexNM achieves up to a 15.94× speedup over the dense systolic-array baseline while requiring 1.47× LUT and 1.54× FF usage relative to the same baseline.