Bias-guided margin loss for robust Visual Question Answering

Abstract

Visual Question Answering (VQA) suffers from language prior issue, where models tend to rely on dataset biases to answer the questions while ignoring the image information. Existing studies have been devoted to mitigating language bias by using extra question-only models or balancing the dataset. However, these works fail to comprehensively identify the bias, despite the fact that some methods utilizing margin loss to separate the biased answer embeddings. In this paper, we propose a bias-guided debiasing architecture with margin loss named as BGML, which utilizes a bias model to guide the margin loss for explicitly locating biases of different question types in the answer space. This distinction of bias prompts the model to avoid the adverse effects of language priors. Additionally, we encourage the bias model to comprehensively learn biases by integrating the adversarial training, knowledge distillation, and contrastive learning. The experimental results show that BGML achieved the state-of-the-art results with 62.28% on VQA-CP v2, while retaining competitive results with 60.84% on VQA v2.