Improving deepfake detection with predictive inter-modal alignment and feature reconstruction in audio–visual asynchrony scenarios

Abstract

Existing multimodal deepfake detection methods primarily rely on capturing correlations between audio–visual modalities to improve detection performance. However, in scenarios such as instant messaging and online video conferencing, network jitter often leads to audio–visual asynchrony, disrupting inter-modal associations and limiting the effectiveness of these methods. To address this issue, we propose a deepfake detection framework specifically designed for audio–visual asynchrony scenarios. First, based on the theory of open balls in metric space, we analyze the variation mechanism of joint features in both audio–visual synchrony and asynchrony scenarios, revealing the impact of audio–visual asynchrony on detection performance. Second, we design a multimodal subspace representation module that incorporates hierarchical cross-modal semantic similarity to address inconsistencies in audio–visual data distributions and representation heterogeneity. Furthermore, we formulate audio–visual feature alignment as an integer linear programming task and employ the Hungarian algorithm to reconstruct missing inter-modal associations. Finally, we introduce a self-supervised masked reconstruction mechanism to restore missing features and construct a joint correlation matrix to measure cross-modal dependencies, enhancing the robustness of detection. Theoretical analysis and experimental results show that our method outperforms baselines in audio–visual synchrony and asynchrony scenarios and exhibits robustness against unknown disturbances.