Spatial navigation of media streams

Abstract

Interactive multimedia walkthrough applications are useful tools for visualizing complex areas. These environments permit navigation through a virtual space based on intuitive actions like "go forward" or "go left". The space is generally constructed using computer graphics models and enhanced with video, still images, and sound. While video is usually incorporated into these models, it is played as taken, and generally as a dependent media, i.e. the navigation controls do not control the video even if the video is a "real" walkthrough of the virtual space. Integrating the real-world media and the computer graphics model by registering both within a common virtual reality framework would allow navigation in the computer graphics model to control video of the corresponding location in the real world and vice versa. This would cause the computer graphics model to be regenerated based on the camera position in the real world video. Additionally, by registering into the common framework, new media data of any type can be added into the presentation. This paper presents a multimedia data structure capable of supporting these operations. A prototypical mobile video capture device is discussed that can record the required video media and process it for inclusion in the framework described.