Structured or Semi-Structured? The Use of Reflection Journals in Postgraduates' Generative Artificial Intelligence Literacy Development in an L2 Academic Writing Context

Generative artificial intelligence (GenAI) not only has the potential to aid L2 academic writing but also poses unique challenges concerning impacts and ethics. Reflection journals, which promote critical thinking and metacognitive awareness, have the capacity to guide GenAI-assisted writing, yet remain underexplored. This study examines and compares the effects of structured reflection journals (SRJs) and semi-structured reflection journals (SSRJs) on enhancing postgraduates' GenAI literacy within an L2 academic writing context. Based on the frameworks proposed by UNESCO and Digital Promise, the study developed a GenAI literacy framework that includes four dimensions: (a) Operational Competence in GenAI Tools, (b) Ethics and Security in GenAI Usage, (c) Critical Evaluation of GenAI Outputs and Autonomy and (d) Reflection in GenAI Application. The study assessed the GenAI literacy levels of 39 participants before and after four GenAI-assisted writing tasks. Reflection journals were completed after each task (20 in the SRJ group and 19 in the SSRJ group). Results revealed significant improvements in GenAI literacy across three dimensions (Dimensions 1 to 3), with no significant difference between the SRJ and SSRJ groups. However, thematic analysis of reflective content showed that SRJs, with their more comprehensive guiding questions, encouraged deeper engagement with GenAI-related rules and more thorough evaluations of GenAI-generated content compared to SSRJs. By comparing different types of reflection journals as effective scaffolds in L2 writing pedagogy, this study encourages the integration of reflective practices into GenAI-assisted L2 academic writing classrooms, aiming to enhance students' critical evaluation skills and ethical awareness.