Mixed reality head mounted displays for enhanced indoor point cloud segmentation with virtual seeds

Abstract

Mixed Reality (MR) Head Mounted Displays (HMDs) offer a hitherto underutilized set of advantages compared to conventional 3D scanners. These benefits, inherent to MR-HMDs albeit not originally intended for such applications, encompass the freedom of hand movement, hand tracking capabilities, and real-time mesh visualization. This study leverages these attributes to enhance indoor scanning process. The primary innovation lies in the conceptualization of manual-positioned MR virtual seeds for the purpose of indoor point cloud segmentation via a region-growing approach. The proposed methodology is effectively implemented using the HoloLens 2 platform. An application is designed to enable the remote placement of virtual tags based on the user's visual focus on the MR-HMD display. This non-intrusive interface is further enriched with expedited tag saving and deletion functionalities, as well as augmented tag visualization through overlaying them on real-world objects. To assess the practicality of the proposed method, a comprehensive real-world case study spanning an area of 330 s² is conducted. Remarkably, the survey demonstrates remarkable efficiency, with 20 virtual tags swiftly deployed, each requiring a mere 2 s for precise positioning. Subsequently, these virtual tags are employed as seeds in a region-growing algorithm for point cloud segmentation. The accuracy of virtual tag positioning is found to be exceptional, with an average error of 2.4 ± 1.8 cm. Importantly, the user experience is significantly enhanced, leading to improved seed positioning and, consequently, more accurate final segmentation results.