Listeners near-optimally integrate acoustic and semantic cues in spoken word recognition: Evidence from experiments manipulating cue order and reliability

Abstract

Understanding spoken language demands integration of linguistic information over time. Previous work has shown that lower-level cues, like acoustic information, and higher-level cues, like semantic information, can be integrated across the course of an utterance in order to categorize words. Questions remain about how these bottom-up and top-down sources are integrated. We propose that a useful approach to this problem is to compare human behavior to normative models of cue integration, such as ideal observer models, which have been successful in other domains. We argue that ideal observer models make three key predictions about how acoustic and semantic information should be integrated during word recognition: (1) additivity: cues have additive effects on categorization; (2) transitivity: cues providing identical information should be treated identically regardless of their temporal order; and (3) reliability weighting: cues should be weighted according to their absolute and relative reliabilities. We conduct two spoken-word recognition experiments in which participants categorize a target word embedded in a sentence. We manipulate acoustic cues on the target word; semantics from the sentence; the cues’ relative timing, and the degree of noise affecting each cue. We find evidence for additivity and transitivity; we also find evidence in favor of absolute reliability driving cue reweighting, but more mixed evidence on the role of relative cue reliability. Overall, our results suggest that listeners are near-optimal in their integration of information across multiple levels of the linguistic hierarchy over time. We also discuss how these results relate to recent error-driven approaches to cue weighting.