Deep learning applied to quad-pixel plenoptic images

Abstract

In recent years, we have seen the development of integrated plenoptic sensors, where multiple pixels are placed under one microlens. It is mainly used by cameras and smartphones to drive the autofocus of the main lens, and it often takes the form of dual-pixels with 2 rectangular sub-pixels. We study the evolution of dual-pixels, the so-called quad-pixel sensor with 2x2 square sub-pixels under the microlens. As it is a simple light field capturing device, we investigate the computational photography abilities of such sensor. We first present our work on pixel-level simulations, then we present a model of image formation taking into account the diffraction by the microlens. Finally, we present new ways to process a quad-pixel images based on deep learning.