Image Inpainting with Context Flow Network

Abstract

Image inpainting using deep and complicated convolutional neural networks(CNN) has recently produced outstanding results. Several researchers have considered employing large receptive fields and deep networks for long-distance information transfer to obtain semantically coherent inpainting results. As a side effect, these strategies would lead to the loss of detail and other artifacts. Motivated by the attention mechanism and sequence-to-sequence model, a novel convolution structure called context flow module is introduced into a coarse-to-fine two stages network, extracting information from distant regions without extra network layers or details loss. The context flow module in the refinement network can effectively gather both spatial and contextual data in the distance, and flow information to the next layer patch by patch. The coarse and refinement networks' backbones are encoder-decoder architecture stacked with gated and dilated convolutions. The refinement network encloses two extra elements: the context flow module and a feature-sharing space. The coarse network generates semantically consistent images with no gaps. The refinement network enhances the sharpness and enriches the details of the initial results. Moreover, a patch-based GAN is applied to stabilize training and generate semantically reasonable results. Experimental results show that our method excels at the performance of the state-of-the-art methods on faces(CelebA), buildings(Paris Street View), and natural images(Places2) datasets. The proposed context flow module can be easily integrated with any existing networks to improve their inpainting performance.