An Efficient Sparse CNN Inference Accelerator With Balanced Intra- and Inter-PE Workload

Sparse convolutional neural networks (SCNNs) which can prune trivial parameters in the network while maintaining the model accuracy has been proved to be an attractive approach to alleviate the heavy computation of convolutional neural networks (CNNs). However, the invalid data resulting from sparse patterns leads to unnecessary and irregular computation workload, which challenges the efficiency of the underlying hardware accelerators. Therefore, this article proposes an SCNN inference accelerator, which can deal with the imbalanced workload both intra- and interprocessing element (PE). A valid weight encoding (VWE) scheme is proposed to compress sparse weights into dense ones to alleviate the load imbalance intra-PE. Leveraging the VWE scheme, a randomized load rearrangement (RLR) method is proposed to dynamically schedule convolution kernels with similar sparsity into the same computation batch to alleviate the load imbalance inter-PEs. In addition, to reduce off-chip memory accesses, a recurrent weight stationary (RWS) dataflow is proposed, which adopts a small-batch and multichannel strategy to stack data from multiple channels within one off-chip access and let them compute simultaneously thereby enabling efficient reuse of on-chip data. Based on the proposed scheme, an efficient SCNN inference accelerator has been designed and verified on the field-programmable gate array (FPGA). Compared with state-of-the-art works, our design achieves $1.16\times $ to $2.77\times $ higher digital signal processors (DSPs) efficiency and $1.75\times $ to $15\times $ higher logic efficiency.