A Study on 4D Light Field Compression Using Multi-focus Images and Reference Views

We propose a novel method of light field compression using multi-focus images and reference views. Light fields enable us to observe scenes from various viewpoints. However, it generally consists of 4D enormous data, that are not suitable for storing or transmitting without effective compression at relatively low bit-rates. On the other hand, 4D light fields are essentially redundant because it includes just 3D scene information. While robust 3D scene estimation such as depth recovery from light fields is not so easy, a method of reconstructing light fields directly from 3D information composed of multi-focus images without any scene estimation is successfully derived. Based on the method, we previously proposed light field compression via multi-focus images as effective representation of 3D scenes. Actually, its high performance can be seen only at very low bit-rates, because there exists some degradation of low frequency components and occluded regions on light fields predicted from multi-focus images. In this paper, we study higher quality light field compression by using reference views to improve quality of the prediction from multi-focus images. Our contribution is twofold: first, our improved method can keep good performance of 4D light field compression at a wider range of low bit-rates than the previous one working effectively only for very low bit-rates; second, we clarify how the proposed method can improve its performance continuously by introducing recent video codec such as HEVC and VVC into our compression framework, that does not depend on 3D-SPIHT previously adopted for the corresponding component. We show experimental results by using synthetic and real images, where quality of reconstructed light fields is evaluated by PSNR and SSIM for analyzing characteristics of our novel method well. We notice that it is much superior to light field compression using HEVC directly at low bit-rates regardless of its light field scan order.