A speech-video synchrony quality metric using CoIA

Abstract

A quality model was built to assess the influence of speech-video asynchrony on the audio-visual quality perception. The audio-visual contents were separated into two categories: “speaker inside” and “speaker outside”, depending on whether the speaker is inside the video. For the first category, speech was shifted in a small scale. DCT and MFCC coefficients were calculated from video and speech separately. A Co-inertia Analysis (CoIA) was used to decide the speech-video correlation, and as the speech progressively shifts, a correlation curve emerged. The curve was modeled by an Gaussian function, and then the function was used to predict the perceptual quality. On the other hand, a Gaussian curve was used to predict the perceptual quality of the “speaker outside” category. A subjective test proved the effectiveness of the proposed method.