Multi-layer background sprite model for 2D-to-3D video conversion

Abstract

This paper presents a technique for semi-automatic 2D-to-3D stereo video conversion, which was known to provide user intervention in segmenting foregrounds and assigning corresponding depth information for key frames and then get the depth maps for other non-key frames via automatic depth propagation. Our algorithm escapes from traditional depth propagation paradigm based on motion estimation and compensation. For foregrounds in non- key frames, object kernels standing for the most confident parts are identified first and then used as the seeds for graph-cut segmentation. Since the graph-cut segmentation for foregrounds is performed independently for each non-key frame, the results will be free of the limitation by objects' motion activity. For backgrounds, all video frames, after foregrounds being removed, are integrated into a common multi-layer background sprite model (ML-BSM) based on image registration algorithm. Users can then draw background depth profiles for the ML-BSM in a video-based manner (not frame-based), thus reducing the human efforts required significantly. Our ML-BSM algorithm is an extension of our prior work, BSM [8], aiming to solve the cases when the foreground and the background have a large depth variation or the camera has a substantial panning/rotating motion. Experiments show that the adoption of multi-layers BSM architecture and iterative foreground refinement based on BSM validation can improve the depth image quality significantly.