Eliminating Non-Overlapping Semantic Misalignment for Cross-Modal Medical Retrieval

Abstract

In recent years, increasing research has shown that fine-grained local alignment is crucial for the cross-modal medical image-report retrieval task. However, existing local alignment learning methods suffer from the misalignment of semantically non-overlapping features between different modalities, which in turn negatively affects the retrieval performance. To address this challenge, we propose a Global-Feature Guided Cross-modal Local Alignment (GFG-CMLA) method. Unlike prior methods that rely on explicit local attention or learned weighting mechanisms, our approach leverages global semantic features extracted from the cross-modal common semantic space to implicitly guide local alignment, adaptively focusing on semantically overlapping content while filtering out irrelevant local regions, thus mitigating misalignment interference without additional annotations or architectural complexity. We validated the effectiveness of the proposed method through ablation experiments on the MIMIC-CXR and CheXpert Plus dataset. Furthermore, comparisons with state-of-the-art local alignment methods indicate that our approach achieves superior cross-modal retrieval performance.