Dynamic feature fusion guiding and multimodal large language model refining for medical image report generation

Abstract

Medical image report generation refers to the automatic generation of text descriptions that correspond to specific medical images. In recent years, the increasing demand for medical imaging from both patients and healthcare institutions has significantly increased radiologists’ workloads. Concurrently, shortages in medical resources and diagnostic capabilities have raised the risks of diagnostic delays and misinterpretations in medical imaging. To alleviate the burden on medical professionals and ensure accurate diagnoses, the task of automated medical report generation has attracted a growing number of researchers. In this context, systems based on deep learning methods combined with general Large Language Models (LLMs) have been developed. However, existing methods face limitations in effectively integrating visual and textual data and they ignore the fact that the contributions of different modalities to diagnostic results vary across cases. Additionally, these approaches fail to address the lack of specialized medical knowledge when applying general LLMs. This paper introduces the Dynamic Feature Fusion Guiding and Multimodal Large Language Model Refining (DFFG-MLLMR) framework, which addresses these limitations through two key components:(1) The DFFG module dynamically adjusts the contributions of visual and textual features based on their diagnostic relevance, ensuring optimal feature utilization for report generation; (2) The MLLMR module integrates visual retrieval methods with fine-tuned LLMs to generate comprehensive and accurate medical reports. Our method achieves quantitatively superior results to other baseline methods on both benchmark datasets. On the IU-Xray dataset, DFFG-MLLMR achieves BLEU-4 of 0.191 and CIDEr of 0.574, exceeding the best conventional approach Token-Mixer. On the MIMIC-CXR dataset, our method achieves BLEU-4 of 0.132 and CIDEr of 0.289, improving upon Token-Mixer by 0.008 and 0.126. Experiments on public datasets demonstrate the superiority of DFFG-MLLMR, showing significant improvements in cross-modal feature fusion performance and enhanced diagnostic quality in automated reports. Furthermore, ablation studies confirm that the DFFG and MLLMR modules contribute complementary improvements, collectively enhancing the accuracy and clinical reliability of reports. The code can be obtained at https://github.com/BearLiX/DFFG-MLLMR.