SRT: Shape Reconstruction Transformer for 3D Reconstruction of Point Cloud from 2D MRI

Abstract

There has been some work on 3D reconstruction of organ shapes based on medical images in minimally invasive surgeries. They aim to help lift visualization limitations for procedures with poor visual environments. However, extant models are often based on deep convolutional neural networks and complex, hard-to-train generative adversarial networks; their problems about stability and real-time plague the further development of the technique. In this paper, we propose the Shape Reconstruction Transformer (SRT) based on the self-attentive mechanism and up-down-up generative structure to design fast and accurate 3D brain reconstruction models through fully connected layer networks only. Point clouds are used as the 3D representation of the model. Considering the specificity of the surgical scene, a single 2D image is limited as the input to the model. Qualitative demonstrations and quantitative experiments based on multiple metrics show the generative capability of the proposed model and demonstrate the advantages of the proposed model over other state-of-the-art models.