Forest before trees? It depends on not only what you see, but also what you hear

Abstract

Prior researches on global–local processing have focused on hierarchical objects in the visual modality, while the real-world involves multisensory interactions. The present study investigated whether the simultaneous presentation of auditory stimuli influences the recognition of visually hierarchical objects. We added four types of auditory stimuli to the traditional visual hierarchical letters paradigm: no sound (visual-only), a pure tone, a spoken letter that was congruent with the required response (response-congruent), or a spoken letter that was incongruent with it (response-incongruent). The data were modeled using a hierarchical drift–diffusion model (HDDM). In Experiment 1, the participants were asked to discriminate the global or local visual letters accompanied by these sounds. Experiment 2 eliminated the global advantage effect by enlarging stimuli to isolate interference mechanisms. Results revealed that response-incongruent speech attenuated both the global advantage effect and robustly reduced the global interference effect across experiments. HDDM analysis demonstrated dual-stage modulation: 1) Perceptual delays during congruent local trials, indicating attentional capture; 2) Decision-stage disruption during local processing, reflecting impaired evidence accumulation for conflict resolution. Critically, only semantically incongruent speech altered decision dynamics, while pure tones affected only perceptual encoding. This study provides new insights into multisensory interactions, showing how auditory stimuli interfere with visual global local perception through attentional filtering mechanisms at different stages of processing.