Spike-driven incepformer: A hierarchical spiking transformer with inception-inspired feature learning

Abstract

Designing spike-based attention mechanisms and Transformer architectures has gradually become a hot topic in the field of Spiking Neural Networks (SNNs). Spiking Transformers typically rely on floating-point and integer computations to achieve performance improvements. However, models that maintain Spike-Driven characteristics, while exhibiting lower energy consumption, often suffer from suboptimal performance. This paper proposes an innovative solution to address this trade-off. Firstly, we introduce feature convolution, expanding the receptive field of attention learning through multi-scale feature connections. Secondly, we design a Spike-Driven Feature Attention (SDFA) mechanism, which significantly reduces computational complexity and enhances performance by utilizing feature matrix operations. Thirdly, we integrate the Inception structure into the Spike-Driven Transformer, replacing traditional MLP layers. Finally, we incorporate diverse branch convolutions to mitigate information loss caused by neurons in the final layer. Experimental results demonstrate that the Spike-Driven Incepformer achieves excellent performance while balancing parameter count and computational cost. On the ImageNet-1k dataset, it attains an accuracy of 80.41%, representing the state-of-the-art for Spike-Driven SNNs. These findings provide new insights for designing low-energy, high-performance spiking neural networks and promote the application of spiking Transformers in broader artificial intelligence domains. Code will be available at https://github.com/2ephyrus/SDIncepformer.