Multi-Models from Computer Vision to Natural Language Processing for Cheapfakes Detection

Abstract

Cheapfakes can compromise the integrity of information and erode trust in multimedia content, making their detection critical. Identifying Out of Context misuse of media is essential to prevent the spread of misinformation and to ensure that news and information are presented accurately and ethically. In this paper, we focus our efforts on Task 1 of the Grand Challenge on Detecting Cheapfakes in ICME2023, which involves detecting triplets consisting of an image and two captions as Out of Context. We propose a new robust approach for detecting Cheapfakes, which are instances of image reuse with different captions. Our proposed approach leverages multi-models in Computer vision and Natural language processing, such as Named entity recognition, Image captioning, and Natural language inference. In our experiments, the proposed multi-models method achieves an impressive accuracy of 78.6%, the highest accuracy among the candidates on the hidden test set. Overall, our approach demonstrates a promising solution for detecting Cheapfakes and safeguarding the integrity of multimedia content. Our source code is public on https://github.com/thanhson28/icme2023.git.