Advanced defense against GAN-based facial manipulation: A multi-domain and multi-dimensional feature fusion approach

Powerful facial image manipulation offered by encoder-based GAN inversion techniques raises concerns about potential misuse in identity fraud and misinformation. This study introduces the Multi-Domain and Multi-Dimensional Feature Fusion (MDFusion) method, a novel approach that counters encoder-based GAN inversion by generating adversarial samples. Firstly, MDFusion transforms the luminance channel of the target image into spatial, frequency, and spatial-frequency hybrid domains. Secondly, we use the specifically adapted Feature Pyramid Network (FPN) to extract and fuse high-dimensional and low-dimensional features that enhance the robustness of adversarial noise generation. Then, we embed adversarial noise into the spatial-frequency hybrid domain to produce effective adversarial samples. Finally, the adversarial samples are guided by our designed hybrid training loss to achieve a balance between imperceptibility and effectiveness. Tests were conducted on five encoder-based GAN inversion models using ASR, LPIPS, and FID metrics. These tests demonstrated the superiority of MDFusion over 13 baseline methods, highlighting its robust defense and generalization abilities. The implementation code is available at https://github.com/LuckAlex/MDFusion.