A mixed-reality museum tourism framework based on HMD and fisheye camera

Mixed-reality (MR) combines virtual and physical components with display device, which can be used to construct new environments where physical and digital objects co-exist and interact in real time. In this paper, we introduce a MR museum tourism system based popular HMD and fisheye camera, which can provide a richer experience by overlying the virtual model in the real environment. First, object detection and tracking methods are used to identify the real object and to obtain its geometrical information from camera. Then, the orientation relationship between real object and camera is decided to convert them to the same coordinate system. Finally, the corresponding model is rendered in the HMD device. Experiments results show that our proposed system can accurately and stably capture the real objects and render the corresponding virtual model in real time.