Day-to-night image translation via transfer learning to keep semantic information for driving simulator

Abstract

Recently, autonomous driving technologies require robust perception performance through deep learning with huge data and annotations. To guarantee performance, perception accuracy should be robust even in nighttime. However, lots of perception technologies perform poorly with nighttime data. It is because most current datasets with annotation are composed of daytime scenes and there are few datasets for adverse conditions especially in nighttime. A massive cost of human resources and time is required to collect large amounts of data with annotation. To deal with the upper problem, many image translation methods by Generative Adversarial Networks (GANs) are proposed to generate realistic synthetic data. However, there is a significant limitation in traditional image translation methods. It is that generated images are inconsistent on semantic information to their original images. To handle this limitation, we propose an image translation with applying transfer learning to keep semantic information. There are two steps to train the proposed network. First, the segmentation network is trained on the source domain, i.e., daytime. After that, we transfer the pretrained segmentation weights to the encoder of generator and retrain only the decoder of GANs for day-to-night image translation. Experimental results show that the proposed method can generate more semantic consistent nighttime images than traditional studies.