Voice in AI-assisted multimodal texts: What do readers pay attention to?

Abstract

Despite the extensive research on voice in traditional text-based writing, there is a notable lack of empirical studies examining this concept within multimodal writing contexts. The shift towards multimodality in writing research, coupled with the rise of Generative Artificial Intelligence (GenAI) in content creation, calls for a deeper understanding of how voice is perceived by readers beyond traditional writing contexts. This mixed-method study addresses this gap by exploring voice construction in GenAI-assisted photo essays from a dialogic perspective. In this study, we invited writing teachers to rank five student-produced photo essays according to their perceived voice strengths and analyzed the rankings using Kendall's Coefficient Concordance. The statistical analysis shows a weak agreement (W = 0.27) among raters, suggesting that voice is perceived quite diversely. The follow-up interviews with six focal raters reveal that they could agree on the importance of having unique ideas and angles in writing, keeping writing coherent and focused, using appropriate quotations, and incorporating images to enhance storytelling. However, opinions diverge regarding using primary and secondary texts, adopting academic discourse features, and including AI-generated images. The study adds to scholarly conversation of voice in composition studies and suggests that divergence in perceiving voice could be leveraged to fuel the discussion about voice in writing pedagogy.