{"title":"Corpus-based insights into multimodality and genre in primary school science diagrams","authors":"Tuomo Hiippala","doi":"10.1177/14703572231161829","DOIUrl":"https://doi.org/10.1177/14703572231161829","url":null,"abstract":"This article presents a data-driven analysis of multimodal genre in a corpus of primary school science diagrams that contains multiple layers of cross-referenced annotations for multimodal discourse structure. The aim is to identify diagram genres in the corpus and describe their multimodal characteristics. To do so, information about expressive resources used in the diagrams and the discourse relations between them is extracted from the corpus, and computer vision is used to approximate the visual appearance of the diagrams. The article also presents a new method for quantifying information about the use of layout space. The resulting description of multimodal discourse structure is processed using UMAP, an unsupervised machine-learning algorithm, in order to identify diagrams that exhibit similar structural characteristics. The analysis allows the identification and characterization of four diagram genres in the corpus, which adopt different rhetorical strategies in combining expressive resources into discourse structures. The analysis also reveals that layout plays a major role in shaping the genre space, which can be further refined using information about the discourse structure. Overall, the results suggest that computational methods can be used to characterize multimodal genre from a bottom-up perspective using low-level information about expressive resources and layout.","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76391148","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Renato Antonio Bertão, Myeong-heum Yeoun, Jaewoo Joo
{"title":"A blind spot in AI-powered logo makers: visual design principles","authors":"Renato Antonio Bertão, Myeong-heum Yeoun, Jaewoo Joo","doi":"10.1177/14703572231155593","DOIUrl":"https://doi.org/10.1177/14703572231155593","url":null,"abstract":"Artificial intelligence is already embedded in several digital tools used across design disciplines. Although it offers advantages in automating and facilitating design tasks, this technology has constraints to empowering practitioners. AI systems steadily incorporate machine learning to deliver meaningful designs but fail in critical dimensions such as creativity. Moreover, the intensive use of AI features to provide a design solution – so-called AI design – challenges the boundaries of the design field and designers’ roles. AI-powered logo makers exemplify a horizon where non-designers can access design tools to create a personal or business visual identity. However, in the current context, these online businesses are limited to randomize layout solutions lacking the visual properties a logo requires. This article reports mixed-method research focusing on AI-powered logo makers’ processes and outcomes. We investigated their capability to deliver consistent logo designs and to what extent their algorithms address logo design principles. Initially, our study identified representative visual principles in logo design-related literature. After probing AI-powered logo makers’ features that enable logo creation, we conducted an exploratory experiment to obtain solutions. Finally, we invited logo design experts to evaluate whether three visual principles (proportion, balance and unity) were incorporated into the layouts. The assessment’s results suggest that these AI design tools must calibrate the algorithms to provide solutions that meet expected logo design standards. Even focusing on a particular AI tool and a few visual principles, our research contributes to initial directions for developing algorithms that embody the complex aspects of visual design syntax.","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76464377","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Catherine Ma, Carol Green, Jinfeng Zhao, V. Egli, T. Clark, Niamh Donnellan, Melody Smith
{"title":"Creative and visual communication of health research: development of a graphic novel to share children’s neighbourhood perspectives of COVID-19 lockdowns in Aotearoa New Zealand","authors":"Catherine Ma, Carol Green, Jinfeng Zhao, V. Egli, T. Clark, Niamh Donnellan, Melody Smith","doi":"10.1177/14703572231157042","DOIUrl":"https://doi.org/10.1177/14703572231157042","url":null,"abstract":"Research dissemination to target stakeholders including communities, policymakers and practitioners is a fundamental element of successful research projects. For many of these stakeholders, however, barriers to access and uptake exist, including time taken to publish, academic jargon, language barriers, paywalled articles and time taken to consume and understand academic outputs. Ultimately these barriers could prevent research from reaching target audiences or could severely delay the uptake of key research messages. Creative and visual dissemination approaches as a complement to traditional academic outputs offer numerous advantages and may improve real-world uptake in a timely manner. In this practitioner piece, the authors present detailed methods for the development of a graphic novel using research findings from an online survey that asked children what they liked about their neighbourhood during COVID-19 lockdowns in Aotearoa New Zealand. Here, they share critical reflections from the process of developing and disseminating this creative communication, with the aim of informing and supporting future creative and visual dissemination of research findings.","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83900823","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Book review: Photographs and the Practice of History","authors":"Linlin Song, Hulin Ren","doi":"10.1177/14703572231161832","DOIUrl":"https://doi.org/10.1177/14703572231161832","url":null,"abstract":"","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135798050","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Book review: Multimodal Literacy in School Science: Transdisciplinary Perspectives on Theory, Research and Pedagogy","authors":"Jing Song, Jinyou Zhou","doi":"10.1177/14703572231157037","DOIUrl":"https://doi.org/10.1177/14703572231157037","url":null,"abstract":"","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-03-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73541943","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Script-switching in Japanese pop culture: a social semiotic multimodal approach","authors":"Y. Matsuda","doi":"10.1177/14703572231155586","DOIUrl":"https://doi.org/10.1177/14703572231155586","url":null,"abstract":"This article explores the nature of multimodal meaning-making in writing by analyzing the script-switching used in Japanese pop culture. Japanese intermixes multiple script systems in one text. Although official and public documents follow standard orthographic conventions, creative works in pop culture texts freely and regularly break these norms by switching script types to create new meanings. However, while many previous works recognize the semiotic effects of script-switching, to this day, there is no systematic account for how and why such meaning-making operates. Notably, script-switching triggers meanings independent of linguistic meaning. Yet, scripts are a visual manifestation of language, so one cannot say they are independent of language. Accordingly, this work attempts to resolve this dilemma by exploring a social semiotic multimodal approach (see Van Leeuwen ‘Typographic meaning’, 2005, and ‘Towards a semiotics of typography’, 2006, and Stöckl ‘Typography: Body and dress of a text’, 2005), which claims the semiotic independence of typography and its interdependency with language. By recognizing the basic unit of the writing system as graphemes, the author argues that the Japanese script systems are categorized into four types of graphemes depending on the linguistic unit they relate to and she demonstrates how the visual unit of the grapheme and the linguistic unit interact in meaning-making. In this analysis, script-switching occurs when grapheme types are intentionally changed to foreground a visual meaning associated with each script system. This article shows that such a meaning reflects the historical development and discursive social practices of writing with multiple script systems throughout the history of the Japanese writing system.","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-02-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81068832","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Fictional mapping: the nature of cartography in film production","authors":"Welby Ings","doi":"10.1177/14703572231152665","DOIUrl":"https://doi.org/10.1177/14703572231152665","url":null,"abstract":"This practitioner piece considers cartographies that permeate and enable the practice of film production. Using his feature film PUNCH (2022) as a case study, Welby Ings discusses ways in which mud maps, overhead maps and relational maps were used as communicative devices in establishing a shared template for understanding the positioning of fictional spaces. Mud maps (see Danesh’s ‘7 pro tips for managing crew parking when filming on location’, 2019), are conventionally used in film production to describe geographies and inform cast and crew about locations. Overhead maps (see Isham’s ‘Creating overhead maps for film directors’, 2017), were employed in association with shot lists, as ‘graphic prompts’ when directing complex scenes. Relational or ‘mental maps’ (see Bays’, ‘Between the scenes: Choosing locations’, 2015) were used in conjunction with site drawings, to establish spatial blueprints for discussing the internal logic of the film’s world with the director of photography, production designers, location scouts, sound recordists, costumers and postproduction collaborators. As the film’s director and cartographer, the author created these maps with a unique visual language that included distinctive considerations of marginalia, typography, line weight, texture, historical allusion and colour, to communicate the emotional tone of the film’s world.","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-02-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86876061","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Book review: Modelling Paralanguage Using Systemic Functional Semiotics: Theory and Application","authors":"Xiaoqin Wu","doi":"10.1177/14703572231152667","DOIUrl":"https://doi.org/10.1177/14703572231152667","url":null,"abstract":"","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-02-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"72480254","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Alien domesticity: representing home during a pandemic.","authors":"Brent Luvaas","doi":"10.1177/14703572221089012","DOIUrl":"https://doi.org/10.1177/14703572221089012","url":null,"abstract":"<p><p>Cultural geography and the related disciplines of urban sociology and anthropology have long focused their theoretical lenses on the city as a space of lived multiplicity. This photographic essay focuses its lens on the home as such a space. During the pandemic lockdown of 2020-2021, many of us spent more time in our homes than we ever had before, working, teaching, schooling, shopping, and barricading ourselves from the outside world. This essay borrows from the often ambiguous and anonymizing aesthetics of street photography to depict the multiple, overlapping worlds of home during a pandemic. Home, as depicted here, is an always unfinished process of affective assemblage and dissolution. The images featured seek to capture that lack of resolution, the messy emotional texture of home life under lockdown.</p>","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9902811/pdf/10.1177_14703572221089012.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10694932","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Walking through the city soundscape: an audio-visual analysis of sensory experience for people with psychosis.","authors":"Sara Merlino, Lorenza Mondada, Ola Söderström","doi":"10.1177/14703572211052638","DOIUrl":"https://doi.org/10.1177/14703572211052638","url":null,"abstract":"<p><p>This article discusses how an aspect of urban environments - sound and noise - is experienced by people walking in the city; it particularly focuses on atypical populations such as people diagnosed with psychosis, who are reported to be particularly sensitive to noisy environments. Through an analysis of video-recordings of naturalistic activities in an urban context and of video-elicitations based on these recordings, the study details the way participants orient to sound and noise in naturalistic settings, and how sound and noise are reported and reexperienced during interviews. By bringing together urban context, psychosis and social interaction, this study shows that, thanks to video recordings and conversation analysis, it is possible to analyse in detail the multimodal organization of action (talk, gesture, gaze, walking bodies) and of the sensory experience(s) of aural factors, as well as the way this organization is affected by the ecology of the situation.</p>","PeriodicalId":51671,"journal":{"name":"Visual Communication","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/67/88/10.1177_14703572211052638.PMC9900689.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10693779","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}