Visualizing prosodic structure: Manual gestures as highlighters of prosodic heads and edges in English academic discourses

Research has shown a close temporal relationship between prominence-lending tonal movements in speech and prominence in manual gesture. However, prosodic structure consists of not only prosodic heads (i.e., pitch accentuation) but also prosodic edges. To our knowledge, no previous studies have assessed the value of prosodic edges (nuclear vs. phrase-initial prenuclear pitch accents) as anchoring sites for different types of gestures (i.e., referential vs. non-referential) while independently controlling for the relative degree of prominence associated with the pitch accent. The English M3D-TED corpus, which contains over 23 minutes of multimodal speech, was analyzed in terms of prosody and gesture. Results showed that while the majority of manual gesture strokes overlapped a pitch accented syllable (85.99%), apex alignment occurred at a relatively low rate (50.4%) and alignment rates did not significantly differ between referential and non-referential gestures. At the phrasal level, crucially our results also showed that strokes align with phrase-initial prenuclear pitch accents over nuclear accents, and this relationship is not driven by relative prominence. These findings show that both prosodic heads and prosodic edges (i.e., phrase initial and final positions) are key sites for both referential and non-referential gesture production.