Grouped top-down reasoning with hierarchical window transformer for visual grounding

Abstract

Visual grounding, which localizes objects in images based on natural language descriptions, requires effective processing of multi-scale visual inputs to capture both fine-grained details and global context for complex and diverse scenes. However, existing transformer-based methods face significant challenges when handling such inputs, including computational complexity that scales quadratically with spatial dimensions and difficulties in effectively aligning cross-scale information. To address these limitations, we propose Grouped Top-Down Reasoning with Hierarchical Window Transformer (GTD-HWT) with two key innovations: (1) a multi-scale input reconstruction strategy that partitions and reconstructs multi-scale inputs into hierarchically structured shorter sequences, effectively preserving both coarse and fine-grained information while reducing computational costs, and (2) a dual multi-head attention mechanism that enables semantic reasoning through parallel inter-window attention for coarse-grained understanding and subsequent intra-window attention for fine-grained refinement guided by coarse-grained priors. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg benchmarks demonstrate that our method achieves significant improvements over state-of-the-art approaches in both referring expression comprehension and segmentation tasks.