M2OCNN: Many-to-One Collaboration Neural Networks for simultaneously multi-modal medical image synthesis and fusion

Abstract

Background and Objective:

Acquiring comprehensive information from multi-modal medical images remains a challenge in clinical diagnostics and treatment, due to complex inter-modal dependencies and missing modalities. While cross-modal medical image synthesis (CMIS) and multi-modal medical image fusion (MMIF) address certain issues, existing methods typically treat these as separate tasks, lacking a unified framework that can generate both synthesized and fused images in the presence of missing modalities.

Methods:

In this paper, we propose the Many-to-One Collaboration Neural Network (M2OCNN), a unified model designed to simultaneously address CMIS and MMIF. Unlike traditional approaches, M2OCNN treats fusion as a specific form of synthesis and provides a comprehensive solution even when modalities are missing. The network consists of three modules: the Parallel Untangling Hybrid Network, Comprehensive Feature Router, and Series Omni-modal Hybrid Network. Additionally, we introduce a mixed-resolution attention mechanism and two transformer variants, Coarsormer and ReCoarsormer, to suppress high-frequency interference and enhance model performance.

M2OCNN outperformed state-of-the-art methods on three multi-modal medical imaging datasets, achieving an average PSNR improvement of 2.4 dB in synthesis tasks and producing high-quality fusion images despite missing modalities. The source code is available at https://github.com/zjno108/M2OCNN.

Conclusion:

M2OCNN offers a novel solution by unifying CMIS and MMIF tasks in a single framework, enabling the generation of both synthesized and fused images from a single modality. This approach sets a new direction for research in multi-modal medical imaging, with implications for improving clinical diagnosis and treatment.