Generative AI Enables Synthesizing Cross-Modality Brain Image via Multi-Level-Latent Representation Learning

Abstract

Multiple brain imaging modalities can provide complementary pathologic information for clinical diagnosis. However, it is huge challenge to acquire enough modalities in clinical practice. In this work, a cross-modality reconstruction model, called fine-grain aware generative adversarial network (FA-GAN), is proposed to reconstruct the target modality images of brain from the 2D source modality images with a dual-stages manner. The FA-GAN is able to mine the multi-level shared latent representations from the source modality images and then reconstruct the target modality image from coarse to fine progressively. Specifically, in the coarse stage, the Multi-Grain Extractor firstly extracts and disentangles the shared latent features from the source modality images, and synthesizes the coarse target modality images with a pyramidal network. The Feature-Joint Encoder then encodes the latent features and frequency features jointly. In the fine stage, the Fine-Texture Generator is fed with the joint codes to fine tune the reconstruction of the fine-grained target modality. The wavelet transformation module is employed to extract the frequency codes and guide the Fine-Texture Generator to synthesize finer textures. Comprehensive experiments from MR to PET images on ADNI datasets demonstrate that the proposed model achieves finer structure recovery and outperforms the competing methods quantitatively and qualitatively.