Corpus-based insights into multimodality and genre in primary school science diagrams

Abstract

This article presents a data-driven analysis of multimodal genre in a corpus of primary school science diagrams that contains multiple layers of cross-referenced annotations for multimodal discourse structure. The aim is to identify diagram genres in the corpus and describe their multimodal characteristics. To do so, information about expressive resources used in the diagrams and the discourse relations between them is extracted from the corpus, and computer vision is used to approximate the visual appearance of the diagrams. The article also presents a new method for quantifying information about the use of layout space. The resulting description of multimodal discourse structure is processed using UMAP, an unsupervised machine-learning algorithm, in order to identify diagrams that exhibit similar structural characteristics. The analysis allows the identification and characterization of four diagram genres in the corpus, which adopt different rhetorical strategies in combining expressive resources into discourse structures. The analysis also reveals that layout plays a major role in shaping the genre space, which can be further refined using information about the discourse structure. Overall, the results suggest that computational methods can be used to characterize multimodal genre from a bottom-up perspective using low-level information about expressive resources and layout.