Multimodal Semantic-Similarity Gate for Real-Time Optimized Deepfake Detection

Abstract

With recent advances in deep learning-based generative artificial intelligence (AI), highly realistic synthetic videos and audio have rapidly proliferated and been increasingly misused, intensifying the need for reliable deepfake detection. Accordingly, growing attention has been given to multimodal AI-based deepfake detection methods. These systems typically analyze the global context of video and audio information using attention mechanisms to detect whether the content is synthetic. However, attention-based multimodal fusion does not distinguish the manipulation status of each modality; therefore, the computational cost increases as the video and audio sequences become longer. Although frame selection can reduce computation, it often causes global structural information loss and omits certain important sequence information. This study proposes a multimodal semantic-similarity gate (MSG)-based deepfake detection framework, which reduces computational cost while preserving the entire video–audio sequence information. The MSG adopts a two-stage backbone to process common sequences of video and audio modalities, comprising a front stage that determines the semantic similarity between modalities and a back stage that controls hard/soft attention. Experimental results show that the proposed MSG maintains an F1-score difference within 0.0001–0.0106 compared with baseline models, while reducing GFLOPs by 10.59–28.07% and per-sample inference latency by 3.35–41.96%. These results demonstrate improved computational efficiency and support real-time deepfake detection.