Language Communication and Multimodal Conversation Analysis

Abstract

According to the “principle of conformity of language expressions,” language users generally convey the meaning of language that is consistent in their language, thoughts, feelings, and appearances, but in ordinary adult language, there are many “violations of the principle of conformity” in which the four elements do not coincide. If the language of the conversation participant is different from the thoughts, feelings, and appearance, it is necessary to process the language data to analyze the participants’ verbal behavior. In order to capture the true state of the on-site discourse, 顾曰国 proposes the construction of a multimodal corpus composed of recorded voices, recorded videos, and transcribed written language materials. In addition, researchers can separately build the corpus required for related research through processing, search, and statistics in a multimodal way. Since this study analyzes dialogue in Chinese dramas, not on-site discourse, the step of recording videos is omitted in the process of collecting language data. It extracts the voice from the recorded video, transcribes the text, and annotates it. Participants use at least dual modalities, such as visual and auditory modalities, and implement language behavior under the influence of other communication factors. Therefore, in the transcription process of written language, the content of written language should be compared with the characteristics of sound language and body expression in the video, and emotional characteristics should also be checked. This study examines the dynamic process of verbal communication, such as how participants with different communication objectives are influenced by what factors are affected during the conversation and what strategies are used to achieve certain results.