Real world objects augmentation in virtual 3D environment RealSense SDK, Deep Learning and Game Engine

Abstract

We propose and demonstrate an approach for real world objects augmentation in a virtual 3D environment using deep neural networks and game engine. The proposed system consists of three basic parts: acquisition, processing and visualization with VR. The processing unit comprises translation of 2D to 3D images of real world objects and augmentation of the real world object in a virtual environment with unity. Through deep learning and RealSense SDK, skeleton data is achieved which helps augment the real world 3D object into a virtual environment in unity. The proposed system enables real-time augmentation of the real world objects in a virtual 3D environment created in unity game engine. Therefore, the system allows users developing mixed reality applications for real life problem solving.