Integration of 3D audio and 3D video for FTV

Abstract

We developed an FTV system to process, and display information of 3D scene in realtime, in which users freely control their own viewpoint/listening-point position. Free listening-point can be generated by either (i) ray-space representation of sound wave field (source sound independent), or (ii) by acoustic transfer function estimation (source sound dependent) and blind separation of sources of sounds. Free viewpoint generation is based on ray-space method, which is enhanced by using multipass dynamic programming. Integration is done by either (i) ray-space representation of sound wave and images together, or (ii) integrating each camera video signal and acoustic transfer function of the same location as integrated 3DAV data. The prototype system of integrated audio-visual viewer achieves both good image and sound qualities in realtime.