Visualizing the Smart Environment in AR: An Approach Based on Visual Geometry Matching

Abstract

This article presents Insight, an AR system for visualizing the IoT-enabled smart environment without relying on the unique appearances, barcodes, world coordinates, or wireless signals of IoT infrastructures. The system analyzes the camera video and motion data taken by mobile AR equipment to extract the self and cross visual geometries describing the poses and geographic distribution of nearby IoT devices. To recognize IoT devices using the extracted geometries, Insight operates in two phases. At deployment time, it learns pairwise mappings from the visual geometries to the corresponding device identities. After that, it leverages the geometries scanned at run time to look for a partial assignment to the recorded geometries, allowing it to automatically recognize the IoT devices in AR view. As such, our system turns the IoT device recognition task into a geometry matching problem, which is further formalized as to perform Subset, Incomplete, and Duplicated Point Cloud Registration (SID-PCR) in this work. We design a deep neural network paying specific edge- and spectral-wise graph attention to solve SID-PCR, and implement a prototype that adaptively requests visual geometry scan and registration operations for accurate recognition. The performance of Insight is validated using both synthetic data and a real-world testbed.