Fine-grained knowledge fusion for retrieval-augmented medical visual question answering

Abstract

Given that medical image analysis often requires experts to recall typical symptoms from diagnostic archives or their own experience, implementing retrieval augmentation in multi-modal tasks like Medical Visual Question Answering (MedVQA) becomes a logical step to facilitate access and use of diverse case data. However, introducing existing retrieval augmentation methods to MedVQA faces two limitations: (1) Due to privacy concerns, direct access to original medical data is typically restricted. (2) The symptoms distinguishing various diseases are often subtle and fine-grained, complicating the task of ensuring that the retrieved information precisely matches the query. To address these challenges, we propose a retrieval augmentation framework with the Fine-Grained Re-Weighting (FGRW) strategy, which employs fine-grained encoding for retrieved multi-source knowledge, avoiding direct access to original image–text data. It then computes re-weighted relevance scores between queries and knowledge, using these scores as supervised priors to guide the fusion of queries and knowledge, thus reducing interference from redundant information in answering questions. Experimental results on PathVQA, VQA-RAD, and SLAKE public benchmarks demonstrate FGRW’s state-of-the-art performance. Code is available at the public repository.¹