Effects of visual gender and frequency shifts on vowel category judgments

Abstract

Visual morphing and a high-quality vocoder were used to study the audio-visual contribution of talker gender to the identification of frequency-shifted vowels. Several acoustic continua from “bit” to “bet” were generated incorporating fundamental frequency (F0) and spectral envelope shifts. Visual continua were constructed from male or female faces, and corresponding steps along the audio/visual continua were synchronized. Boundary shifts emerged for both acoustic cues (F0 and spectral envelope shifts) and visual cues (visual gender) as predicted by the co-variation of these properties in natural speech. Results suggest that perceivers exploit learned relationships between acoustic and visual cues when judging vowel identity.