MG-SSAF: An advanced vision Transformer

Abstract

Despite the excellent performance of local-window-based multi-head self-attention (MSA), non-overlapping windows hinder cross-window feature interaction, leading to high computational complexity. This paper presents MG-SSAF, a novel Vision Transformer backbone addressing these problems. First, we propose a Space-wise Separable Multi-head Self-attention (SS-MSA) mechanism to reduce the computational complexity further. Then, an extra Attention Fusion Module (AF Module) is introduced for the attention weights in SS-MSA to enhance the representation ability of the similarity. Next, we present a Multi-scale Global Multi-head Self-attention (MG-MSA) method to perform the global feature interaction. Moreover, we propose to perform the window-based MSA and the global MSA simultaneously in one attention module to realize local feature modeling and global feature interaction. The experimental results demonstrate that the MG-SSAF achieves superior performance with fewer parameters and lower computational complexity. The code is available at https://github.com/shuaiyang11/MG-SSAF.