Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

Abstract

Communicative hand gesticulations are tightly coupled to prosodic aspects of speech of speech. Psychologists have characterized this multimodal synchrony as a preplanning process governed by a cognitive timing mechanism acquired only later in development. However, it has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impetus on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) uttered consonant-vowel CV (/pa/) mono-syllables in rhythmic fashion while moving the upper limbs (or not). We show that respiration-related movement is affected by (especially high-impetus) gesturing when vocalizations occur near peaks in physical impetus. We further show that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, we find tight relations between respiration-related movement and vocalization, even in the absence of movement, and even more strong respiration-acoustic relations are found when upper-limb movement is present. The current findings expand a developing line of research showing that speech acoustics is modulated by functional biomechanical linkages between hand gesture and the respiratory system.