GTR: Global Token Retrieval for Window-Based Vision Transformer

Abstract

Vision transformers (ViTs) have achieved remarkable success in visual recognition tasks due to their strong capability to model long-range dependencies through global self-attention. However, the inherent quadratic complexity of global attention severely limits their scalability when processing high-resolution inputs. To address this issue, window-based attention mechanisms have been proposed to reduce computational overhead, but they often struggle with limited receptive fields and insufficient global context modeling. In this work, we propose global token retrieval vision transformer (GTR-ViT), a novel and efficient ViT framework that integrates a global token retrieval (GTR) mechanism with an enhanced feed-forward network (FFN) in a synergistic manner. The GTR mechanism dynamically retrieves globally relevant tokens for each token, enriching the local features with comprehensive global context. These globally enhanced features are then fed into a gated feed-forward network equipped with depthwise convolution, which further refines local representations through adaptive feature modulation. In turn, the improved local features serve as more representative queries for the GTR module, enabling more accurate GTR in subsequent layers. This mutual reinforcement forms a virtuous cycle of local-global interaction, significantly boosting the model’s capability in both context understanding and feature discrimination. Extensive experiments on image classification and downstream tasks across multiple datasets demonstrate that GTR-ViT consistently surpasses prior state-of-the-art methods, highlighting the effectiveness of this collaborative design.