Speech-reading on the lips as a cognitive resource to understand speech in noise.

Abstract

In challenging acoustic scenarios, speech processing is often linked to listening effort, which can be described as the balance between cognitive demands and motivation to understand speech. In such conditions, people usually rely on several behavioral strategies to support speech understanding and reduce listening effort (e.g., speech-reading behavior). Still, it is not clear what cognitive mechanisms underlie the use of behavioral strategies for listening. We hypothesized that the cognitive and motivational dimensions of listening effort may also drive speech-reading strategies spontaneously adopted in challenging conditions. Normal-hearing adults (N = 64) performed audiovisual speech-recognition in noise, in combination with a concurrent mnemonic task with low vs. high working memory engagement to set cognitive demands. Motivation was manipulated between-subjects through fixed or performance-related monetary rewards. Speech-reading was tracked with eye-movement, and pupil dilation served as a physiological measure of listening effort, confirming manipulation effectiveness. We found that exerted listening effort intensifies speech-reading behavior, with motivation playing a key role in this behavioral adaptation to enhanced cognitive demands. These findings document the association between internal mental processes and behavioral adaptation in the speech domain.