GFSNet: Gaussian Fourier with sparse attention network for visual question answering

Abstract

Visual question answering (VQA), a core task in multimodal learning, requires models to effectively integrate visual and natural language information to perform reasoning and semantic understanding in complex scenarios. However, self-attention mechanisms often struggle to capture multi-scale information and key region features within images comprehensively. Moreover, VQA involves multidimensional and deep reasoning about image content, particularly in scenarios involving spatial relationships and frequency-domain features. Existing methods face limitations in modeling multi-scale features and filtering irrelevant information effectively. This paper proposes an innovative Gaussian Fourier with Sparse Attention Network (GFSNet) to address these challenges. GFSNet leverages Fourier transforms to map image attention weights generated by the self-attention mechanism from the spatial domain to the frequency domain, enabling comprehensive modeling of multi-scale frequency information. This enhances the model’s adaptability to complex structures and its capacity for relational modeling. To further improve feature robustness, a Gaussian filter is introduced to suppress high-frequency noise in the frequency domain, preserving critical visual information. Additionally, a sparse attention mechanism dynamically selects optimized frequency-domain features, effectively reducing interference from redundant information while improving interpretability and computational efficiency. Without increasing parameter counts or computational complexity, GFSNet achieves efficient modeling of multi-scale visual information. Experimental results on benchmark VQA datasets (VQA v2, GQA, and CLEVR) demonstrate that GFSNet significantly enhances reasoning capabilities and cross-modal alignment performance, validating its superiority and effectiveness. The code is available at https://github.com/shenxiang-vqa/GFSNet.