Unsupervised Contrastive Refinement With Graph-Aware Multimodal Interaction for Radiology Report Generation

Abstract

Automatic Radiology Report Generation (RRG) reduces medical workload by automatically generating diagnostic reports from X-ray images. In recent studies, an improved performance has been observed by first classifying the image and then incorporating clinical outcomes into the RRG pipeline, with cross-modal interactions between image and text. However, these studies have limitations (specifically on chest X-ray), including : (1) inconsistent cross-modal learning due to under-utilization of textual information, and (2) reliance on noisy classification labels, resulting in inaccurate attention and invariant report generation. To address these issues, a Medical Context-aware RRG Framework (MedCARF) is proposed, which introduces a Disease-Aware Visual Textual Alignment (D-ViTAlign) module to capture organ–abnormality relationships from scene graphs and visual features. Label refinement is achieved using an unsupervised Augmentation Distillation with Contrastive Refinement (AuDiCoR) loss, which iteratively refines the visual features via augmentation alignment and loss modelling. In addition, the knee OAI dataset is benchmarked for the RRG task, and the performance of MedCARF contributions is explicitly tested on this dataset. Furthermore, Knee RRG performance is evaluated using multimodal information, including biomechanical data. Experimental results showed that a minimum 3.2% improvement in METEOR and a 4.22% AUC gain over classification SOTA are achieved for chest X-ray, while a 9.69% average improvement across all NLG metrics is observed for knee RRG with biomechanical data integration. Overall, contextually accurate reports are generated, and consistent classification performance is achieved, including for minority classes, demonstrating the robustness and clinical applicability of the proposed framework. The Code is available at: https://github.com/adaydar/MedCARF.