Deciphering the Practices and Affordances of Different Reasoning Strategies through Multimodal Learning Analytics

Multimodal analysis has had demonstrated effectiveness in studying and modeling several human-human and human-computer interactions. In this paper, we explore the role of multimodal analysis in the service of studying complex learning environments. We use a semi-automated multimodal method to examine how students learn in a hands-on, engineering design context. Specifically, we combine, audio, gesture and electro-dermal activation data from a study (N=20) in which students were divided into two experimental conditions. The two experimental conditions, example-based reasoning and principle-based reasoning, have previously been shown to be associated with different learning gains and different levels of design quality. In this paper we study how the two experimental conditions differed in terms of their practices and processes. The practices included four common multimodal behaviors, that we've entitled ACTION, TALK, STRESS and FLOW. Furthermore, we show that individuals from the two experimental conditions differed in their usage of the four common behavior both on aggregate, and when we model their sequence of actions. Details concerning the data, analytic technique, interpretation and implications of this research are discussed.