Associating video with related documents

Abstract

Reflecting the increasing importance of handling multimedia data, many studies are made on indexing to TV broadcast video. Multimedia data consist of image, audio, and text, and various research on analysis of each individual medium has been made. Especially, image processing has been the main issue when handling multimedia for a long time. But recently, it has started to be considered that image processing alone is insufficient for thorough understanding of multimedia data. In the 1990’s, integrated processing that supplements the incompleteness of information from each medium has become a trend. Following this trend, we are trying to integrate TV programs with related documents, taking advantage of the relative easiness of extracting semantic structures from text media. Among various programs, cultural programs are considered as appropriate sources since (1)supplementary documents are available and (2)the video contains a lot of implicit information that integration could be helpful to thorough understanding of supplementary texts. Many attempts have been made to index video by means of multimedia integration. But sufficient accuracy for practical use is not necessarily achieved since their subjects are too general to achieve accuracy from elemental technoiogies by making use of domain specific characteristics. In our method, we examine and construct a practical system using relatively simple elemental technologies by reflecting the result of one medium’s process to another. We will focus on cooking programs, so that we can take advantage of domain specific constraints and knowledge. Through the examination in this specific domain, and the usage of a supplementary document and its analysis, we aim for proposing a novel advanced multimedia integration method. Using the result of this method, we also propose an integrative restructuring method of the multimedia data provided both from the video and the supplementary document.