An Energy-Efficient FPGA Accelerator for Swin Transformer

Recently, transformers have shown strong performance in tasks such as computer vision and natural language processing. Notably, Swin Transformer has gained significant attention for its low computational complexity and impressive performance in computer vision tasks, due to its window attention mechanism and hierarchical architecture. However, these features also make hardware deployment more complicated. In this brief, we present an energy-efficient field-programmable gate array (FPGA) accelerator for Swin Transformer to support the hierarchical architecture and execute the window attention. First, we introduce a systolic array with alterable datapath (SAAD) to conduct the window attention. Second, we split the patch merging operation and design a data rearrangement module, which reduces the computing latency induced by the data rearrangement in Swin Transformer. Third, we present a parallelized dual-array dataflow to support different computing operations in Swin Transformer. We implement the accelerator on the Xilinx XCZU19EG platform. The proposed architecture achieves a throughput per digital signal processing (DSP) of 0.630 giga operations per second (GOPS)/DSP, which is $1.94\times $ higher than existing works.