Automatic behaviour analysis reveals links between facial expression and speech acts’ illocutionary point, strength, and valence

Abstract

There is abundant literature indicating that, beyond its role in emotional communication, facial behaviour could be essential in the way individuals carry out performative functions of speech. Using speech acts theory, we formalised three ways facial signals could relate to verbal utterances: 1) by authenticating the illocutionary point of statements, 2) by authenticating the valence of the mental states underlying speech acts, and 3) by supporting attempts at modulating the strength of a speech act's illocutionary force. We investigated these relationships in an observational case study of a public hearing of the U.S. Senate. Overall, facial behaviour was more intense during performative than during declarative speech acts and we observed a negative relationship between facial expressivity and the illocutionary strength of speech acts. Facial behaviour was also related to the valence of speech acts, albeit to a lesser extent. While we observed multiple links between individual facial Action Units and speech acts, the pattern of results does not indicate a unique set of associations between specific facial movements and specific properties of speech acts. The present study supports the idea that facial behaviour plays a role in the pragmatics of social interaction in that it could be used by speakers to show the authenticity of their communicative intentions and to modulate the illocutionary force of verbal statements. It also suggests that the cognitive and emotional states underlying the production of speech acts could share structural and functional elements with the appraisal processes involved in the production of facial behaviour.