A Neuromorphic Transformer Architecture Enabling Hardware-Friendly Edge Computing

Abstract

The transformer model has demonstrated significant capabilities in various intelligent tasks, attracting widespread attention in recent years. However, it involves numerous complex operations, including large-bit-width multiplication, division, matrix transposition, and exponentiation. These require substantial storage and computational resources, making it challenging to deploy on edge devices. This work introduces a neuromorphic transformer architecture with low hardware cost for AI edge computing (AI-EC). At the structural level, it absorbs scaling factors within the self-attention mechanism into weight matrixes, thereby eliminating the division caused by the scaling operation. Additionally, a transposition calculation method is proposed to perform matrix transposition using dedicated memory access strategies and optimized data flow designs, which reduces logic resource overhead and avoids memory access discontinuities. At the computing paradigm level, the architecture employs spike-driven computing, substituting multi-bit multipliers with AND logic for synaptic operations. The paradigm introduces high sparsity to computational data, which is effectively exploited to reduce the computational workload of the architecture. The results indicate that the architecture successfully eliminates high-cost operators and significantly reduces computational expenses. Eventually, this architecture is verified as a prototype using a 28 nm CMOS process library, demonstrating a compact logic area of sub-0.2 mm² and a high energy efficiency of 0.34 pJ/SOP @ 50MHz. This work is expected to promote the application of transformers in edge computing and the development of intelligent edge applications.