Bimodal PET/MRI generative reconstruction based on VAE architectures.

Abstract

•Objective:In this study, we explore positron emission tomography(PET)/magnetic resonance imaging (MRI) joint reconstruction within a deeplearning (DL) framework, introducing a novel synergistic method. •Approach:We propose a new approach based on a variational autoencoder (VAE)constraint combined with the alternating direction method of multipliers (ADMM)optimization technique. We compare several VAE architectures, including jointVAE, mixture of experts (MoE) and product of experts (PoE), to determine theoptimal latent representation for the two modalities. We trained then evaluatedthe architectures on a brain PET/MRI dataset. •Main results:We showed that our approach takes advantage of each modalitysharing information to each other, which results in improved peak signal-to-noiseratio (PSNR) and structural similarity (SSIM) as compared with traditionalreconstruction methods, particularly for short acquisition times. We find that theone particular architecture, MMJSD, is the most effective for our methodology. •Significance:The proposed method outperforms classical approaches especiallyin noisy and undersampled conditions by making use of the two modalities together to compensate for the missing information.