Benchmarking vision-language models for brain cancer diagnosis using multisequence MRI

The rapid adoption of Large Language Models (LLMs) in various fields has prompted growing interest in their application within the healthcare ecosystem. In particular, Vision Language Models (VLMs) offer potential for generating radiology reports. This study aims to perform a comprehensive evaluation of state-of-the-art VLMs with varying sizes and domain specializations to determine their effectiveness in generating radiology reports for MRI scans of brain cancer patients. We conducted a comparative analysis of several open-source VLMs, including Qwen2-VL, Meta-Vision 3.2, PaliGemma 2, DeepSeek-VL2, Nvidia open-source models, and medical-specific VLMs. Each model family was assessed across small, medium, and large variants. A benchmark dataset comprising multisequence brain MRI scans of cancer patients was curated with expert annotation and guidance from board-certified radiologists. The models were evaluated on their ability to generate complete radiology reports, using objective metrics, reasoning models (R1 and o1 models that judged the AI generations based on completeness, conciseness, and correctness), and human experts. Model performance varied significantly across size and type. Among large-scale models, Meta-Vision 3.2 90B achieved the highest scores with o1 of 70.19% and R1 of 68.09%. In the medium category, Meta-Vision 11B and DeepSeek-VL-2-27B outperformed others, achieving o1 scores of 57.56% and 53.44%, and R1 scores of 51.06% and 52.75%, respectively. For smaller models, Qwen2-VL-2B demonstrated the best performance (o1: 23.88%, R1: 23.25%). To our knowledge, this is the first study evaluating VLMs for generating comprehensive radiology reports for brain cancer diagnosis using multisequence MRI. Our findings reveal substantial performance differences based on model size and specialization, offering important guidance for future development and optimization of medical VLMs to support diagnostic radiology workflows.