Medical Multimodal Image Transformation With Modality Code Awareness

Abstract

In the planning phase of radiation therapy, positron emission tomography (PET) images are frequently integrated with computed tomography (CT) and MRI to accurately delineate the target region for treatment. However, obtaining additional CT or magnetic resonance (MR) images solely for localization purposes proves to be financially burdensome, time-intensive, and may increase patient radiation exposure. To alleviate these issues, a deep learning model with dynamic modality translation capabilities is introduced. This approach is achieved through the incorporation of adaptive modality translation layers within the decoder module. The adaptive modality translation layer effectively governs modality transformation by reshaping the data distribution of features extracted by the encoder using switch codes. The model’s performance is assessed on images with reference images using evaluation metrics, such as peak signal-to-noise ratio, structural similarity index measure, and normalized mean square error. For results without reference images, subjective assessments are provided by six nuclear medicine physicians based on clinical interpretations. The proposed model demonstrates impressive performance in transforming nonattenuation corrected PET images into user-specified modalities (attenuation corrected PET, MR, or CT), effectively streamlining the acquisition of supplemental modality images in radiation therapy scenarios.