Magic Touch: Interacting with 3D Printed Graphics

Graphics like maps and models are important learning materials. With recently developed projects, we can use 3D printers to make tactile graphics that are more accessible to blind people. However, current 3D printed graphics can only convey limited information through their shapes and textures. We present Magic Touch, a computer vision-based system that augments printed graphics with audio files associated with specific locations, or hotspots, on the model. A user can access an audio file associated with a hotspot by touching it with a pointing gesture. The system detects the user's gesture and determines the hotspot location with computer vision algorithms by comparing a video feed of the user's interaction with the digital representation of the model and its hotspots. To enable MT, a model designer must add a single tracker with fiducial tags to a model. After the tracker is added, MT only requires an RGB camera, so it can be easily deployed on many devices such as mobile phones, laptops and smart glasses.