Facilitating learners' self-assessment during formative writing tasks using writing analytics toolkit

Abstract

Background

Learners' writing skills are critical to their academic and professional development. Previous studies have shown that learners' self-assessment during writing is essential for assessing their writing products and monitoring their writing processes. However, conducting practical self-assessments of writing remains challenging for learners without help, such as formative feedback.

Objectives

To facilitate learners' self-assessment in writing, we developed a writing analytics toolkit and used data visualisation and cutting-edge machine learning technology that provides real-time and formative feedback to learners.

Methods

To investigate whether our newly-developed tool affects the accuracy and process of learners' self-assessment, we conducted a lab study. We assigned 59 learners to complete writing (2 h) and revising (1 h) tasks. During the revision stage, we randomly assigned the learners to two groups: one group used the writing analytics toolkit while the second group was not granted access to the toolkit. Learners' self-assessment accuracy and process of self-assessment were compared between the two groups.

Results

In our study, we found the toolkit helped learners in the experimental group improve the self-assessment accuracy of their writing products compared to the learners in the control group. In addition, we also found that the affordances of the toolkit affected the learners' self-assessment process, and poor design affordances may have prevented the learners from reflecting by themselves.

Conclusions

Together, our empirical study shed light on the design of future writing analytics tools which aim at improving learners' self-assessment during formative writing processes.