A structural model of semiotic alignment: The classification of multimodal ensembles as a novel machine learning task

Abstract

In addition to the well-known linguistic alignment processes in dyadic communication — e.g., phonetic, syntactic, semantic alignment — we provide evidence for a genuine multimodal alignment process, namely semiotic alignment. Communicative elements from different modalities “routinize into” cross-modal “super-signs”, which we call multimodal ensembles. Computational models of human communication are in need of expressive models of multimodal ensembles. In this paper, we exemplify semiotic alignment by means of empirical examples of the building of multimodal ensembles. We then propose a graph model of multimodal dialogue that is expressive enough to capture multimodal ensembles. In line with this model, we define a novel task in machine learning with the aim of training classifiers that can detect semiotic alignment in dialogue. This model is in support of approaches which need to gain insights into realistic human-machine communication.