Segmentation-enhanced Medical Visual Question Answering with mask-prompt alignment using contrastive learning and multitask object grounding

Abstract

Medical Visual Question Answering (MedVQA) aims to provide clinical suggestions by analyzing medical images in response to textual queries. However, existing methods struggle to accurately identify anatomical structures and pathological abnormalities, leading to unreliable predictions. Many deep learning-based approaches also lack interpretability, making their diagnostic reasoning opaque. To address these challenges, this paper proposes Mask-Prompt Aligned Visual Question Answering (MPA-VQA), a two-stage framework that integrates segmentation information into the MedVQA process. First, a segmentation model is trained to detect key structures within medical images. To mitigate the issue of limited segmentation annotations, this paper introduces an improved CutMix-based data augmentation strategy. Second, segmentation masks are used to generate prompts, which are incorporated into the question-answering process for the first time to enhance interpretability. Third, to improve the alignment between image, mask, and prompt representations, this paper proposes a dual-granularity mask-prompt alignment (MPA) method. At the image level, MPA employs contrastive learning to encourage global consistency, while at the object level, it leverages multi-task object grounding to enhance localization accuracy. A mask-guided attention mechanism is also introduced to ensure the model focuses on clinically relevant image regions. Finally, the proposed MPA-VQA is validated on the SLAKE and MedVQA-GI datasets, demonstrating state-of-the-art performance. Notably, MPA-VQA improves location-related question accuracy by 6.37% on MedVQA-GI. MPA-VQA is also a plug-and-play framework that can be seamlessly integrated into existing MedVQA architectures without requiring major modifications.