Advancing image inpainting efficiency: An exploration of pixel and channel split operations

Abstract

Deep learning-based image inpainting techniques have achieved unprecedented results using encoder–decoder structures to recover complex missing areas of an image. Recent inpainting models use additional information or networks (e.g., landmarks, edges, styles, and filters) to realize improved restoration performance, but at the cost of increased computational resources. To improve the relationship between inpainting performance and the number of model parameters, researchers have investigated efficient structural approaches such as recurrent and residual connection structures. However, these methods are difficult to apply in the general encoder–decoder structure. In this study, we explored the downsampling and upsampling operations associated with an encoder–decoder structure. We propose two novel split operations: the pixel-split operation (PSO) and channel-split operation (CSO). The proposed PSO transfers image features from high to low resolution with two dilation rate effects and a similar number of parameters as existing downsampling operations. Conversely, the proposed CSO increases the image resolution using only one-fourth the number of parameters of existing upsampling operations. The restoration performance and efficiency of the proposed model were evaluated in terms of five metrics on public datasets, e.g., the Places2 and CelebA datasets, to validate our proposed operations’ contribution to inpainting performance. We achieved state-of-the-art performance and reduced the size of the parameters by 20%. An ablation study was conducted to confirm the effect of each operation on the CelebA-HQ dataset. Results indicated that these split operations exhibit an advanced relationship between inpainting performance and optimization of the model parameters. The corresponding codes are available online (https://github.com/MrCAIcode/Split_operation_for_inpainting).