Fully Quantized Matrix Arithmetic-Only BERT Model and Its FPGA-Based Accelerator

Abstract

In this paper, we propose a fully quantized matrix arithmetic-only BERT (FQ MA-BERT) model to enable efficient natural language processing. Conventionally, the BERT model relies on floating point arithmetic for inference and requires not only linear matrix multiplication but also nonlinear functions, such as softmax and normalization functions, which significantly increase hardware costs. In contrast, the proposed FQ MA-BERT model quantizes all activations and weights to 8-bit integer precision and ternary precision, respectively, i.e., the proposed model is fully quantized. Moreover, all nonlinear layers are replaced with matrix arithmetic operations. This means that the proposed model consists solely of integer-precision linear matrix arithmetic operations, allowing a significant reduction in hardware resources. To validate the hardware-friendliness of the proposed FQ MA-BERT model, we implement an accelerator for the proposed model on AMD Zynq Ultrascale+ FPGA. The proposed accelerator comprises only two types of integer-precision matrix multiplication units without the need for specialized circuits to perform nonlinear functions. The implementation results show that the hardware resource efficiency of the proposed accelerator is 1.8 times higher than that of conventional FPGA-based BERT accelerators.