Prototyping interaction with everyday artifacts: training and recognizing 3D objects via Kinects

In this paper we explore and prototype the interaction with everyday passive objects. We present an approach for 3D object training and recognition, leveraging Kinect sensors: the Dominant Orientation Templates (DOT) method allows real-time object training and multiple Kinects speed up the training process by learning the object from multiple viewpoints simultaneously with the object background removed. A proof-of-concept usage scenario, 3D real-time Lego building instruction, has been developed based on this approach: the system learns the individual Lego pieces and Lego building steps in advance; the users thus construct the Lego model with 3D visual hints attached to present Lego pieces.