Res2former: A multi-scale fusion based transformer feature extraction method

Abstract

In this paper, we propose Res2former, a novel lightweight hybrid architecture that combines convolutional neural networks (CNNs) and Transformers to effectively model both local and global dependencies in visual data. While Vision Transformer (ViT) demonstrates strong global modeling capability, it lack locality and translation-invariance, making it reliant on large-scale datasets and computational resources. To address this, Res2former adopts a stage-wise hybrid design: in shallow layers, CNNs replace Transformer blocks to exploit local inductive biases and reduce early computational cost; in deeper layers, we introduce a multi-scale fusion mechanism by embedding multiple parallel convolutional kernels of varying receptive fields into the Transformer’s MLP structure. This enables Res2former to capture multi-scale visual semantics more effectively and fuse features across different scales. Experimental results reveal that with the same parameters and computational complexity, Res2former outperforms variants of Transformer and CNN models in image classification (80.7 top-1 accuracy on ImageNet-1K), object detection (45.8 $A{P}^{box}$ on the COCO 2017 Validation Set), and instance segmentation (41.0 $A{P}^{mask}$ on the COCO 2017 Validation Set) tasks. The code is publicly accessible at https://github.com/hand-Max/Res2former.