SMR-agents: Synergistic medical reasoning agents for zero-shot medical visual question answering with MLLMs

Existing medical visual question answering (Med-VQA) systems often lack transparent reasoning and robustness, limiting their clinical reliability. This study proposes the Synergistic Medical Reasoning Agents (SMR-Agents) framework to address these limitations by simulating collaborative consultation among multidisciplinary medical expert agents, thereby enhancing interpretability and diagnostic reliability. SMR-Agents first constructs a structured medical scene graph from the input image and question to identify and highlight relevant visual features. A pre-trained large language model then acts as a general practitioner, automatically selecting and coordinating a team of specialized medical expert agents based on this scene graph. The recruited experts engage in iterative reasoning: domain-specific diagnostic agents generate initial answer hypotheses, and a group of consulting experts conducts a peer-review discussion to refine the answer and formulate its explanatory rationale. The entire process operates in a zero-shot manner without task-specific training of the models. Evaluation on three public Med-VQA datasets and a private colorectal image dataset demonstrates that SMR-Agents achieves state-of-the-art performance across all benchmarks. Notably, it yields significant improvements in accuracy for open-ended questions and produces more interpretable reasoning compared to existing methods. These results demonstrate that combining structured scene understanding with iterative multi-expert collaboration substantially enhances both the accuracy and transparency of Med-VQA systems. The SMR-Agents framework thus provides a robust, interpretable approach to AI-assisted medical diagnosis, aligning machine reasoning with the expert-driven consultation processes used in clinical practice.