Joint exploitation of multiple media from multimedia to databases

Abstract

Multimedia content analysis offers many exciting research opportunities and is a necessary step towards automatic understanding of the content of digital documents. Digital documents are typically composite. Processing in parallel and integrating low-level information computed over each of the media that compose a multimedia document can yield knowledge that stand-alone and isolated analysis could not discover.Joint processing of multiple media is very challenging, even at the lowest analysis levels. Coping with imperfect synchronization of pieces of information, mixing extremely different kinds of information (numerical or symbolic descriptions, values describing intervals or instants, probabilities and distances, HMM and Gaussians, ...), and reconciling contradictory outputs are some of the obstacles which make processing of multimedia documents much more difficult than it seems at first glance.This talk will first show what may be gained from jointly analyzing multimedia documents. It will then briefly overview the typical information that can be extracted from major media (video, sound, images and text) before focusing on the problems that arise when trying to use all this information together. We hope to convince researchers to start trying to solve these problems, since they directly hamper the acquisition of higher-level knowledge from multimedia documents.