Digital facial augmentation for interactive entertainment

Abstract

Digital projection technology allows for effective and entertaining spatial augmented reality applications. Leveraging the capabilities of reasonably accurate object tracking using commodity cameras and/or depth sensors to determine the 3D position and pose of objects in real time, it is possible to project dynamic graphical content on arbitrary surfaces, such as a person's face. Coupling these capabilities with a simple drawing application, participants can have the experience of “painting” on someone's face, or even on their own, by observing the projection in a mirror. Similarly, integrating 2D rigid-body, fluid and gravity simulation, one may interact with virtual objects projected on their own face or body.