BETA: A Bit-Grained Transformer Attention Accelerator With Efficient Early Termination

Abstract

Attention-based large language models (LLMs) have revolutionized the natural language processing (NLP). Despite their impressive effectiveness, the quadratic complexity of self-attention incurs heavy computational and memory burdens. Dynamic sparse attention techniques emerge as a solution, however, the introduced extra prediction stage, coupled with costly data memory access, diminishes their hardware efficiency. To address these limitations, this brief proposes BETA, a fine-grained algorithm-architecture co-design tailored for sparse attention. First, a bit-grained multi-round filter (BMF) prediction is proposed to unveil and eliminate redundant memory access hidden in the sparsity prediction stage. Second, an adaptive and lightweight max-based threshold selection (MTS) strategy is developed to work in concert with the bit-wise prediction process. Third, a bit-wise out-of-order execution (BOOE) scheme is employed to enhance hardware utilization during bit-wise prediction. Finally, an elaborate architecture is designed to translate the theoretical complexity reduction into practical performance improvement. Implementation results demonstrate that BETA achieves $5.4\times $ , $6.5\times $ , $1.8\times $ improvements in energy efficiency than the state-of-the-art Transformer accelerators Sanger, Spatten and SOFA, respectively, while maintaining comparable inference accuracy.