Learned Insignificance of Credibility Signs

A large part of how people learn about their shared world is via social information. However, in complex modern information ecosystems, it can be challenging to identify deception or filter out misinformation. This challenge is exacerbated by the existence of a dual-learning problem whereby: (1) people draw inferences about the world, given new social information; and simultaneously (2), they draw inferences about how credible various sources of information are, given social cues and previous knowledge. In this context, we investigate how social influence and individual cognitive processing interact to explain how one might lose the ability to reliably assess information. Crucially, we show how this happens even when individuals engage in rational belief updating and have access to objective cues of deception.

Using an agent-based model, the Reputation Game Simulation, we show that mere misinformation is not the problem: The dual-learning problem can be solved successfully with limited Bayesian reasoning, even in the presence of deceit. However, when certain agents consistently engage in fully deceptive behavior, intentionally distorting information to serve nonepistemic goals, this can lead nearby agents to unlearn or discount objective cues of credibility. This is an emergent delusion-like state, wherein false beliefs resist correction by true incoming information. Further, we show how such delusion-like states can be rehabilitated when agents who had previously lost the ability to discern cues of credibility are put into new, healthy—though not necessarily honest—environments.

Altogether, this suggests that correcting misinformation is not the optimal solution to epistemically toxic environments. Though difficult, socially induced cognitive biases can be repaired in healthy environments, ones where cues of credibility can be relearned in the absence of nonepistemic communication motives.