Multimodality and Hypertext: Theoretical and Empirical Considerations

Abstract

One important contribution commonly ascribed to hypertext is the ability to combine different forms of expression, and so be considered 'multimodal' (or, at least, 'multimedial'). On closer analysis, however, theorizing just what this entails has remained limited. Similarly to the situation that long held concerning 'written' texts, it is too easily assumed that different modalities, sometimes labelled with terms such as 'text' or 'image', combine 'naturally' and so users should be able to follow such combinations with relative ease. Research on literacy, particularly with respect to contemporary media configurations, has shown this assumption to be false. Constructing coherent interpretations of combinations of modalities can be far from straightforward, even when supported by good interface design; with poor design, which from the perspective of displayed 'documents' is unfortunately rather common, finding intended interpretations can present significant challenges. Now, when translated to the even more complex medial environment of hypertext, these potential problems are magnified considerably. Moreover, traditional considerations of where the 'boundaries' of hypertext might lie are now being redrawn as hypertext and the increasingly 'hyper'-connected medial world become increasingly permeable. The entire multimodal world of social media and participatory digital cultures might then be considered from a hypertext perspective, but research on hypertext itself lacks conceptual tools with the power necessary to engage with that world. Simple 'extensions' of traditional notions of hypertext are likely to prove insufficient for a full-blown account of multimodality. In this talk we address these concerns from the perspective of current developments in multimodality studies, where the starting point is communication as such, regardless of the expressive forms that are used for that communication and whether communication is mediated computationally, via interlinked artefacts and pathways, or by cross-linked practices of digital and non-digital use. In short, current medial practices demand that hypertext be seen not simply as, for example, a shift from page-based documents to video, but as a further computationally supported environment for the development and deployment of core multimodal theoretical constructs such as semiotic modes, media and genres. We introduce these concepts and show several practical examples of processing from ongoing projects with a variety of media.