ERP correlates of semantic inconsistencies in deepfakes

Abstract

Advances in deepfake technology raise concerns about disinformation spread. Novel deepfake technologies make it increasingly difficult to distinguish between real and fake media content. The current study investigated how speakers’ credibility, the participants’ traits and attitudes may influence the brain processing of audio deepfakes arguing for and against COVID-19 vaccination. We analyzed the electroencephalograms (EEGs) of 61 participants who supported or opposed COVID-19 vaccination. The participants were exposed to audio deepfakes portraying two speakers—a prominent medical doctor (pro-vaccine advocate) or a prominent COVID-19 dissident (anti-vaccine advocate)—making statements that were congruent or incongruent with their publicly known stances. Сritical words that contradicted the medical doctor's (pro-vaccine advocate) public opinion elicited a stronger delayed N400-like response with a latency of 500–750 ms compared to the critical words that matched his public opinion. We observed no similar effect for the critical words of the popular actress (anti-vaccine advocate). The speaker's credibility was significantly predicted by the amplitude of the N400 component to critical words that contradicted speakers’ public opinion, while participants’ intentions to share deepfakes were predicted by their neural responses to critical words that matched speakers’ public opinion. Our results do not only support previous behavioral findings that information is differently processed depending on source credibility but link them to the brain mechanisms of semantic processing of deepfakes.