Adaptive bias learning via gradient-based reweighting and constrained pruning for robust Visual Question Answering

Abstract

Visual Question Answering (VQA) presents significant challenges in cross-modal reasoning due to susceptibility to dataset biases, spurious correlations, and shortcuts learning, which undermine model robustness. While ensemble methods mitigate bias via joint optimization of a bias model and a target model during training, their efficacy remains limited by suboptimal bias exploitation and model capacity imbalances. To address this, we propose the Adaptive Bias Learning Network (ABLNet), a novel framework that systematically enhances bias capture for improved generalization. Our approach introduces two key innovations: (1) Gradient-driven sample reweighting, which quantifies per-sample bias magnitude via training gradients and prioritizes low-bias samples to refine bias model training; (2) Constrained network pruning, deliberately restricting bias model capacity to amplify its focus on bias patterns. Extensive evaluations on VQA-CPv1, VQA-CPv2, and VQA-v2 benchmarks confirm our ABLNet’s superiority, demonstrating generalizability across diverse question types. The code will be released at https://github.com/runminwang/ABLNet.