Colorful 3d reconstruction from a single image based on deep learning

Abstract

Simultaneously recovering the 3D shape and its surface color from a single image has been a very challenging. In this paper, we substantially improve Soft Rasterizer that is a state-of-the art method for 3D color object reconstruction. The model adopts the structure of the encoder and decoder with a single image as input. Firstly, the features are extracted by the encoder, and then they are simultaneously sent to the shape generator and the color generator to obtain the shape estimate and the corresponding surface color, and finally the final colorful 3D model is rendered by the differentiable renderer. In order to ensure the details of the reconstructed 3D model, this paper introduces an attention mechanism into the encoder to further improve the reconstruction effect. For surface color reconstruction, we propose a combination loss. The experimental results show that compared with the 3D reconstruction network models 3D-R2N2 and OccNet, the intersection-over-union (IOU) increases by 10% and 3% in our model. Compared to the open source project SoftRas_O, the model increases by 3.8% on structural similarity (SSIM) and decreases by 1.2% on mean square error (MSE).