Do passive cross-modal validation processes occur when processing multimedia materials?

Background

In text comprehension research, a passive validation mechanism has been observed that checks the consistency between incoming and previous text information (or prior knowledge).

Aims

In two pre-registered online studies, we investigated whether a passive cross-modal validation mechanism occurs during the processing of multimedia materials (i.e., text combined with pictures).

Samples

Participants (Experiment 1: N = 146; Experiment 2: N = 235) were recruited via Prolific.

Methods

We used the epistemic Stroop paradigm (Richter et al., 2009), which makes use of the fact that the passive validation mechanism induces positive or negative response tendencies that can interfere with the processing of an unrelated task if it requires an opposite response. Participants received either matching (valid) or mismatching (invalid) text-picture stimuli. Following each stimulus, participants performed an unrelated probe-word task reacting to the probe words “wrong” or “right”. The dependent variables were reaction time and error rates in the unrelated probe-word task. Experiment 1 used one-sentence-picture stimuli, while Experiment 2 used longer text-segments-picture stimuli.

Results

Linear mixed-effects models showed interactions of validity and probe word for reaction times (Experiments 1 & 2) and error rates (Experiment 1). Post-hoc comparisons indicated prolonged reaction times or higher error rates when the probe word task required a response opposite to the outcome of the validation process.

Conclusions

This study is the first to demonstrate that a passive cross-modal validation mechanism checks the consistency between written text and accompanying pictures. We discuss the implications of these findings for understanding information processing in multimedia contexts.