Diverse Information Aggregation with Adaptive Graph Construction and prompts for deepfake detection

Abstract

Due to the misuse of face manipulation techniques, there has been increasing attention on deepfake detection. Recently, some methods have employed ViTs to capture the inconsistency in forged faces, providing a global perspective for exploring diverse and generalized patterns to avoid overfitting. These methods typically divided an image into fixed-shape patches. However, each patch contains only a tiny fraction of facial regions, thereby inherently lacking explicit semantic and structural relations with other patches, which is insufficient to model the global context information effectively. To enhance the global context interaction, a Diverse INformation Aggregation (DINA) framework is proposed for deepfake detection, which consists of two information aggregation modules: Adaptive Graph Convolution Network (AGCN) and Multi-Scale Prompt Fusion (MSPF). Specifically, the AGCN utilizes a novel strategy to construct neighbors of each token based on spatial and feature relations. Then, a graph convolution network is applied to aggregate information from different tokens to form a token with rich semantics and local information, termed the group token. These group tokens can be used to form robust representations of global information. Moreover, the MSPF utilizes prompts to incorporate unique forgery traces from complementary information, i.e., multi-scale and frequency information, into group tokens in a fine-grained and adaptive manner, which provides extra information to further improve the robustness of group tokens. Consequently, our model can learn robust global context-aware representations, capturing more generalized forgery patterns from global information. The proposed framework outperforms the state-of-the-art competitors on several benchmarks, showing the generalization ability of our method.