Multimedia retrieval through spatio-temporal activity maps

As multiple video cameras and other sensors generate very large quantities of multimedia data in media productions and surveillance applications, a key challenge is to identify the relevant portions of the data and to rapidly retrieve the corresponding sensor data. Spatio-temporal activity maps serve as an efficient and intuitive graphical user interface for multimedia retrieval, particularly when the media streams are derived from multiple sensors observing a physical environment. We formulate the media retrieval problem in this context, and develop an architecture for interactive media retrieval by combining spatio-temporal "activity maps" with domain specific event information. Activity maps are computed from trajectories of motion of objects in the environment, which in turn are derived automatically by analysis of sensor data. We present an activity map based video retrieval system for the sport of tennis and demonstrate that the activity map based scheme significantly helps the user in a) discovering the relevant portions of the data, and b) non-linearly retrieving the corresponding media streams.