GAN-Based Bayer Pattern Image Synthesis for Raw Domain Computer Vision Tasks

Abstract

In this paper, we introduce a novel approach for converting established color image datasets into their RAW counterparts using generative adversarial networks (GANs), aiming to tackle the challenge of obtaining RAW training data for RAW domain computer vision tasks, such as computer vision-oriented ISP pipeline design, RAW domain denoise, etc. We provide theoretical analysis to demonstrate how leveraging transformed data during GAN training can bolster the model’s capacity to learn the underlying distribution of the original data. The RAW Bayer pattern images can therefore be generated by configuring the transformations in the ISP pipeline into the GAN’s training process. Evaluation results show that RAW images generated by our method outperform the existing methods in the downstream computer vision tasks, and the training process of the proposed method is more stable. Our experiments show that the characteristics of the synthesized images are similar to those of the real images. By training denoising and object detection frameworks using the synthesized RAW Bayer images, we showcase that they can be used in end-to-end vision system (from RAW images to vision tasks with ISP in the loop) with negligible performance degradation.