Adults interpret iconicity in speech and gesture via the same modality-independent process.

Abstract

Iconicity is the resemblance or similarity between the form of a signal and its meaning. In two studies, we investigated whether adults interpret iconicity in speech and gesture via a modality-independent mechanism (Study 1, N = 40; Study 2, N = 348). Participants in both studies completed two verb-action matching tasks. In these tasks, either spoken verbs or co-speech gestures were manipulated at different rates to iconically represent actions (i.e., manners of locomotion) performed at corresponding rates. We hypothesized that adult participants would map verbs spoken at a fast rate to fast actions and verbs spoken at a slow rate to slow actions. Similarly, we hypothesized that adult participants would map gestures produced with faster hand movements to fast actions and gestures produced with slower hand movements to slow actions. Moreover, we predicted a positive correlation between the two task performances. In both a pilot study (Study 1) and a preregistered replication (Study 2), participants successfully used iconic cues to match verbs to actions in both verb-action matching tasks. Crucially, adult participants who performed well in the verb-action matching task with iconic speech cues also performed well in the verb-action matching task with iconic gesture cues. This weak positive correlation remained significant even after controlling verbal working memory performance. These findings suggest that adults may rely on a modality-independent cognitive mechanism for interpreting iconicity both within and across different sensory modalities, using both visual information from gestures and auditory information from speech to identify referent actions.