A Transformer utilizing bidirectional cross-attention for multi-modal classification of Age-Related Macular Degeneration

Age-related Macular Degeneration (AMD) ranks among the leading causes of blindness globally, especially among people over 50. Color fundus photograph (CFP) and optical coherence tomography (OCT) B-scan image are both widely applied in diagnostic phase of AMD. However, most existing multimodal approaches are based on traditional convolutional neural networks (CNN), which usually have limited local receptive fields when processing cross-modal information. Therefore, we propose a model that integrates CNN and transformer architectures for the diagnosis of AMD. Specifically, we first extract features through CNN to learn local representations of the images, and then employ bidirectional cross attention blocks with intramodal and intermodal attention. This allows the model to learn global representations from all input modalities, enabling it to capture long-range dependencies and enhance multimodal feature fusion through its global modeling capabilities. Moreover, we apply data augmentation for effective training. Our data augmentation approach leverages class activation mapping (CAM) as a conditional input to guide a GAN-based network in synthesizing high-resolution CFP and OCT images. Extensive experiments were conducted were carried out on a publicly available AMD dataset to assess the effectiveness of our model. Our method achieves an F1-score of 0.897 and an accuracy of 84.3% on the test set. The results indicate that our proposed approach significantly outperforms multiple baselines for multimodal AMD diagnosis.