VCF: An effective Vision-Centric Framework for Visual Question Answering

Abstract

Recently, the wide application of large language models in the field of Visual Question Answering(VQA) has significantly boosted the progress in this field. Despite achieved advancements, LLMs cannot fully perceive and comprehend visual information well from image. Therefore, how to fully mine visual information is very important for language models to effectively deal with the VQA task. In response to this challenge, we propose a straightforward yet effective Vision-Centric Framework(VCF) for VQA, which mainly includes an adaptive visual perceptron module, a multi-source feature fusion module, and a large language model. The adaptive visual perceptron module effectively condenses and integrates the extensive visual information sequence from the visual encoder output using a fixed number of query embeddings. The multi-source feature fusion module is concentrated on extracting fine-grained visual perception information by fusing visual features of different scales. Finally, by channeling their outputs, the language model leverages its extensive implicit knowledge to produce a more nuanced and precise synthesis of visual information, ultimately delivering the answer. The synergy and complementarity of the two modules jointly enhance the robustness of the model. Through extensive experiments, VCF achieves nearly state-of-the-art experimental results on datasets such as VQAv2, OK-VQA, GQA, Text-VQA and others. At the same time, a series of ablation experiments have been conducted to demonstrate the efficacy of the proposed module. Additionally, VCF even achieves better or equivalent performance compared to some larger-scale models, such as LLaVa-1.5, Pink.