SFormer: An end-to-end spatio-temporal transformer architecture for deepfake detection

Abstract

Growing AI advancements are continuously pacing GAN enhancement that eventually facilitates the generation of deepfake media. Manipulated media poses serious risks pertaining court proceedings, journalism, politics, and many more where digital media have a substantial impact on society. State-of-the-art techniques for deepfake detection rely on convolutional networks for spatial analysis, and recurrent networks for temporal analysis. Since transformers are capable of recognizing wide-range dependencies with a global spatial view and along temporal sequence too, a novel approach called “SFormer” is proposed in this paper, utilizing a transformer architecture for both spatial and temporal analysis to detect deepfakes. Further, state-of-the-art techniques suffer from high computational complexity and overfitting which causes loss in generalizability. The proposed model utilized a Swin Transformer for spatial analysis that resulted in low complexity, thereby enhancing its generalization ability and robustness against the different manipulation types. Proposed end-to-end transformer based model, SFormer, is proven to be effective for numerous deepfake datasets, including FF++, DFD, Celeb-DF, DFDC and Deeper-Forensics, and achieved an accuracy of 100%, 97.81%, 99.1%, 93.67% and 100% respectively. Moreover, SFormer has demonstrated superior performance compared to existing spatio-temporal and transformer-based approaches for deepfake detection.