Efficient 3D magnetic resonance image reconstruction by 2D transformers and attention-based fusion model

Abstract

Utilizing 3D networks for reconstructing images from undersampled 3D k-space data shows potential in accelerating 3D MR imaging. CNN-based reconstruction network U-Net is such a classical method and is widely used in accelerating MRI research. However, directly reconstructing 3D MR images using 3D U-Net, due to the huge amount of 3D MRI data and high computational complexity of 3D convolution, results in significant memory consumption. To address the dependence of 3D MRI reconstruction on high computing resources, we propose an efficient 3D MRI Slice-to-Volume and Fusion Reconstruction (SVFR) method, which reduces the memory consumption requirements by 40% compared to 3D U-Net. Specifically, the proposed method integrates attention-based reconstruction and fusion models into a unified framework. For saving computational resources, instead of directly processing 3D data, we select 2D undersampled slices from three mutually orthogonal directions, and introduce the pretrained 2D Vision Transformer into MRI reconstruction field, reconstructing 3D MR images from 2D slices. In addition, for compensating the loss of spatial details between adjacent slices caused by the process of reconstructing slices, we employ a volume-wise fusion model to extract deep features of reconstructed 3D MR images along original three directions and fuse them on a spatial level, preserving finer spatial details. The experimental results on large 3D multi-coil brain k-space dataset and Stanford Fullysampled 3D FSE Knees dataset clearly demonstrate that the proposed method exhibits excellent reconstruction performance and efficiency under various accelerations.