People indexing in TV-content using lip-activity and unsupervised audio-visual identity verification

Abstract

Our goal is to structure TV-content by person allowing a user to navigate through the sequences of the same person. To let a user browse through the content without restriction on people within it, this structuration has to be done without any pre-defined dictionary of people. To this end, most methods propose to index people independently by the audio and visual information, and associate the indexes to obtain the talking-face one. Unfortunately, this approach combines clustering errors provided in each modality. In this work, we propose a mutual correction scheme of audio and visual clustering errors. First, the clustering errors are detected using indicators suspecting a talking-face presence. Then, the incorrect label is corrected according to an automatic modification scheme. Two modification schemes are proposed and evaluated : one based on systematic correction of the a priori supposed less reliable modality while the second proposes to compare unsupervised audio-visual models scores to determine which modality failed. Experiments on a TV-show database show that the proposed correction schemes yield significant improvement in performance, mainly due to an important reduction of missed talking-faces.