ELAFormer: Early Local Attention in multi-scale vision transFormers

Abstract

Vision Transformers have demonstrated remarkable success in vision tasks and have shown great potential when compared to CNN-based models. However, Transformers tend to prioritize the global context and overlook the local features between patches. Recent studies suggest that initializing the relative position between query and key tokens can limit attention distance, allowing for effective attention to local features without using convolutional blocks, similar to convolutional kernels. Based on this insight, this paper proposes a new hybrid multi-scale model called Efficient Local Attention transFormer (ELAFormer). In this model, we propose a Window-based Positional Self-Attention (WPSA) module that focuses on adjacent tokens for short-distance features when querying the key token. Furthermore, we improve the conventional Spatial Reduction Attention (SRA) module by employing Depth-wise Separable (DS) convolution instead of standard down-sampling convolution(DSSRA) for long-distance contexts. By stacking these two modules, extensive experiments demonstrate that our model, with a small size of only 28M, achieves 82.9% accuracy on ImageNet classification with an input size of 224 × 224. Our model outperforms state-of-the-art Transformer models. The small ELAFormer model surpasses the tiny focal transformer by +1.3% mAP with RetinaNet 1x on COCO and +1.8/+2.0% mIoU/MS mIouU with UperNet on ADE20k, serving as a strong backbone for the most challenging computer vision tasks.