FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models

Abstract

Recent advancements in using FPGAs as co-processors for language model acceleration, particularly for energy efficiency and flexibility, face challenges due to limited memory capacity. This limitation hinders the deployment of transformer-based language models. To address this challenge, we propose a novel software-hardware co-optimization framework that integrates Hessian-based intra-layer mixed-precision quantization with a runtime bit-configurable FPGA accelerator. Our proposed Hessian-based row-wise weight quantization addresses hardware inefficiencies in traditional parameter-wise and channel-wise approaches by enabling mixed-precision weight matrices to be split into two uniform-precision matrices, thereby simplifying hardware requirements. Additionally, our Query-Key coupled attention activation quantization optimally aligns precision within each outer product pair in attention calculations, reducing hardware complexity and memory management overhead. Our concurrent quantization method balances the benefits of row-wise weight quantization and Query-Key coupled activation quantization while maximizing energy efficiency through multi-precision optimization. To support this algorithm, we design a multi-precision FPGA accelerator capable of handling both 2n-based and non-2n mixed-precision operations. It is implemented on a single Xilinx ZCU102 FPGA board, operating at 200MHz with a power consumption of 15.08W during inference on the 110-million-parameter BERT-Base and 345-million-parameter GPT-2 Medium transformer models. Coupled with the proposed algorithm and dataflow optimization, it enables on-chip storage of all necessary parameters, minimizing off-chip memory access. Experimental results demonstrate that our FPGA accelerator significantly outperforms existing solutions, achieving energy efficiency improvements ranging from $2.22\times $ to $17.23\times $ over state-of-the-art FPGA accelerators.