DII-FRSA: Diverse image inpainting with multi-scale feature representation and separable attention

Abstract

Diverse image inpainting is the process of generating multiple visually realistic completion results. Although previous methods in this area have seen success, they still exhibit some limitations. First, one-stage approaches must make a trade-off between diversity and consistency. Second, while two-stage approaches can overcome such problems, they require autoregressive models to estimate the probability distribution of the structural priors, which has a significant impact on inference speed. This paper introduces DII-FRSA, a method for diverse image inpainting utilizing multi-scale feature representation and separable attention. In the first stage, we build a Gaussian distribution from the dataset to sample multiple coarse results. To enhance the modeling capability of the Variational Auto-Encoder, we propose a multi-scale feature representation module for the encoder and decoder. In the second stage, the coarse results are refined while maintaining overall consistency of appearance. Additionally, we design a refinement network based on the proposed separable attention to further improve the quality of the coarse results and maintain consistency in the appearance of the visible and masked regions. Our method was tested on well-established datasets-Places2, CelebA-HQ, and Paris Street View, and outperformed modern techniques. Our network not only enhances the diversity of the completed results but also enhances their visual realism.