Intelligent Multimodal Analysis Framework for Teacher-Student Interaction

Abstract

This paper constructed a multi-modal analysis framework of teacher-student interaction based on intelligent technology. Voiceprint recognition was used to divide the teaching video into slices according to sentences and then used speech recognition, speech emotion analysis, gaze point estimation, and other technologies to recognize and encoded the multimodal behavior of each slice. We analyzed 10 lessons using the event sampling method proposed in the analysis framework in comparison with the classical temporal sampling analysis method and demonstrated the results of multimodal interaction analysis of an instructional video as an example. The results indicated that the event sampling method proposed not only reduces the number of analysis units but also has more complete information about the utterance of each unit, overcoming the incomplete information or information redundancy of analysis units caused by the mechanical segmentation of temporal sampling. The multimodal analysis showed that taking into account both teacher-student verbal and nonverbal interactions can reveal richer and deeper information about classroom teaching and learning. This framework provides an important reference for intelligent multimodal analysis of teacher-student interaction.