Multi-Task Collaboration for Cross-Modal Generation and Multi-Modal Ophthalmic Diseases Diagnosis

Multi-modal diagnosis of ophthalmic disease is becoming increasingly important because combining multi-modal data allows for more accurate diagnosis. Color fundus photograph (CFP) and optical coherence tomography (OCT) are commonly used as two non-invasive modalities for ophthalmic examination. However, the diagnosis of each modality is not entirely accurate. Compounding the challenge is the difficulty in acquiring multi-modal data, with existing datasets frequently lacking paired multi-modal data. To solve these problems, we propose multi-modal distribution fusion diagnostic algorithm and cross-modal generation algorithm. The multi-modal distribution fusion diagnostic algorithm first calculates the mean and variance separately for each modality, and then generates multi-modal diagnostic results in a distribution fusion manner. In order to generate the absent modality (mainly OCT data), three sub-networks are designed in the cross-modal generation algorithm: cross-modal alignment network, conditional deformable autoencoder and latent consistency diffusion model (LCDM). Finally, we propose multi-task collaboration strategy where diagnosis and generation tasks are mutually reinforcing to achieve optimal performance. Experimental results demonstrate that our proposed method yield superior results compared to state-of-the-arts.