3D-VRVT: 3D Voxel Reconstruction from A Single Image with Vision Transformer

Abstract

Deep CNN methods have shown very competitive performance in 3D voxel reconstruction from single-view synthetic clean-background images. However, how to generate the target object from a real-world image with clutter background is rarely studied. In this paper, we present a novel network named 3D-VRVT for 3D voxel reconstruction from a single image. Unlike pure CNN-based methods in the past, our 3D-VRVT extracts region features with Vision Transformer (ViT) encoder based on self-attention mechanism, and then a well-designed voxel decoder is used to generate three-dimensional voxel from the encoded image features. The experimental results show that our 3D-VRVT can reconstruct 3D voxel from both synthetic clean-background and real-world images effectively.