Utility of Gestural Cues in Indexing Semantic Miscommunication

Abstract

In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as \emph{miscommunications} in conversations. While detecting these miscommunications is generally easy for the speakers as well as for observers, it is not currently understood which cues contribute to their detection and to what extent. We investigate the possibility of indexing the occurrence of miscommunications in psychotherapeutic face-to-face conversations from gestural patterns. The applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in the conversations. Both simple and complex classifiers are constructed using different features taken from the gesture data. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there \emph{does not exist a single gestural feature} that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than short-term ones.