Dual-way magnetic resonance image translation with transformer-based adversarial network

Background

The magnetic resonance (MR) image translation model is designed to generate MR images of required sequence from the images of existing sequence. However, the generalization performance of MR image generation models on external datasets tends to be unsatisfactory due to the inconsistency in the data distribution of MR images across different centers or scanners.

Purpose

The aim of this study is to propose a cross-sequence MR image synthesis model that could generate high-quality MR synthetic images with high transferability for small-sized external datasets.

Methods

We proposed a dual-way magnetic resonance image translation model using transformer-based adversarial network (DMTrans) for MR image synthesis across sequences. It integrates a transformer-based generative architecture with an innovative discriminator design. The shifted window-based multi-head self-attention mechanism in DMTrans enables efficient capture of global and local features from MR images. The sequential dual-scale discriminator is designed to distinguish features of the generated images at multi-scale.

Results

We pre-trained DMTrans model for bi-directional image synthesis on a T1/T2-weighted MR image dataset comprising 4229 slices. It demonstrates superior performance to baseline methods on both qualitative and quantitative measurements. The SSIM, PSNR, and MAE metrics for synthetic T1 images generation based on T2 images are 0.91 ± 0.04, 25.30 ± 2.40, and 24.65 ± 10.46, while the metric values are 0.90 ± 0.04, 24.72 ± 1.62, and 23.28 ± 7.40 for the opposite direction. Fine-tuning is then utilized to adapt the model to another public dataset with T1/T2/proton-weighted (PD) images, so that only 6 patients of 500 slices are required for model adaptation to achieve high-quality T1/T2, T1/PD, and T2/PD image translation results.

Conclusions

The proposed DMTrans achieves the state-of-the-art performance for cross-sequence MR image conversion, which could provide more information assisting clinical diagnosis and treatment. It also offered a versatile and efficient solution to the needs of high-quality MR image synthesis in data-scarce conditions at different centers.